Quantcast
Channel: SPAN blogs
Viewing all articles
Browse latest Browse all 15

Processing large datasets from laptop at home

$
0
0

I have been working on processing lots of large datasets, such as long-term remote sensing data, and large-scale sensor network data. It is often necessary to log in a remote server and process these data for a long time. However, if you start a program from your laptop and then want to do other things, your program will be killed if you log out the server, or lose internet connection, or close your laptop lid to sleep mode. So you want to leave your program running in background on the remote server even if you log out the system.
A simple way to do that in linux is to use the "nohup" command. The following is the detailed procedure that I use to run my matlab or python codes on a remote server from my laptop at home.

1. log in a server via ssh: ssh user_name@lab1-1.eng.utah.edu, for University of Utah CADE Lab users.

2. write a simple shell script, like run_mycode.csh,
for Python code, it can be as follows:
ipython << EOF
run mycode.py
exit
EOF>

for Matlab code:
matlab << EOF
mycode
exit
EOF>

3. excecute the shell script depending on which script (csh/sh/tcsh) used in the server: nohup csh run.csh & for C shell.

4. check if your program is running, then close the terminal.

Note that, if you find something wrong with your Python or Matlab code, you can always kill that process, make changes to your code, and then repeat the above procedure.

Another way to process large datasets from home without worrying about network connection and logout problems is to use shell session management tool "Screen", which is available for most Linux distributions like Red Hat and Ubuntu. For Ubuntu servers, you can also use "Byobu", which is a more advanced version of Screen.


Viewing all articles
Browse latest Browse all 15

Trending Articles