Introduction

In standard usage of the cluster, you will not interact directly with the compute nodes. Submitting, cancelling, and status checking of jobs is done through SLURM commands like sbatch and squeue. Occasionally it is useful to interact directly with the compute node, for example to check the status or memory usage for an individual process, to to conduct a computationally intensive matlab session. This page will show how to do that.

Even though the capability exists to run interactive jobs on the compute nodes, this is often not the best way to work. It is usually preferred to run computations as batch (non-interactive) jobs. We encourage users to contact us so that we can discuss the problem and determine the best approach.

Running interactive jobs on compute nodes

It is possible to use a compute node interactively. This may be accomplished by issuing the following command on the front end node

[araim1@maya-usr1 ~]$ salloc -n1 srun -n1 -N1 --pty --preserve-env $SHELL
salloc: Granted job allocation 102560
[araim1@n1 ~]$ squeue
  JOBID PARTITION     NAME     USER ST       TIME  NODES QOS     NODELIST(REASON)
 102560   develop     srun   araim1  R       0:59      1 normal  n1
[araim1@n1 ~]$

Notice that the prompt has changed to indicate that we are useing node n1 in the develop queue. A call to the squeue command shows that we are indeed inside of the SLURM job. We can now launch MATLAB, R, or other interactive tools, and the computations will run on node n1 within our job

[araim1@maya-usr1 ~]$ salloc -n1 srun -n1 -N1 --pty --preserve-env $SHELL
salloc: Granted job allocation 102560
[araim1@n1 ~]$ matlab
...
[araim1@n1 ~]$ R
...
[araim1@n1 ~]$ exit
salloc: Relinquishing job allocation 102560

Logging out of the shell will end the job

Note that the usual limitations on memory, time usage, etc are in effect for interactive jobs.

We ask that users avoid "squatting" on compute nodes - opening long interactive sessions without actively using them - since this ties up cluster resources. And please make sure to exit the session when you are finished.

It is possible to specify options for the interactive jobs. To request a batch node rather than a develop node:

[araim1@maya-usr1 ~]$ salloc --partition=batch -n1 srun -n1 -N1 --pty --preserve-env $SHELL

To raise the time limit:

[araim1@maya-usr1 ~]$ salloc --time 00:25:00 -n1 srun -n1 -N1 --pty --preserve-env $SHELL

To get exclusive access to the node:

[araim1@maya-usr1 ~]$ salloc --exclusive -n1 srun -n1 -N1 --pty --preserve-env $SHELL

The full set of options for salloc can be seen by running the usual "man salloc".

Gathering diagnostics for a processes in a SLURM job

Suppose we've got the following job already running on the cluster. In this example, it's a parallel R job running on the two nodes of the develop queue, and using 8 processes per node.

[araim1@maya-usr1 ~]$ squeue
  JOBID PARTITION     NAME     USER  ST       TIME  NODES QOS    NODELIST(REASON)
  59512   develop test-job   araim1   R       0:01      2 normal n[1-2]
[araim1@maya-usr1 ~]$

Using the following syntax, we can run the "hostname" command on all nodes corresponding to our job.

[araim1@maya-usr1 ~]$ srun --jobid 59512 hostname
n1
n2
[araim1@maya-usr1 ~]$

Note that these new tasks execute within the same "job containers" as the already-running task 59512. The new tasks draw from the same resources such as time and memory limits, and number of allocated CPU cores. Therefore, it is recommended that this mechanism only be used for small and infrequent auxiliary jobs, to avoid interfering with the main job.

A more useful application would be to run a "ps" on each node, showing important information about our processes such as status, memory and CPU usage, etc. Note that in the output below, some lines were very long so they have been truncated.

[araim1@maya-usr1 ~]$ srun --jobid 59512 ps u -u araim1
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
araim1   12400  0.0  0.0  40748  1852 ?        S    14:05   0:00 /usr/cluster/openmpi/1.3.3-p1/gcc/4/bin/orted ...
araim1   12402 99.2  0.2 381676 50368 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12403 99.2  0.2 360664 51112 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12404 99.2  0.2 315640 49884 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12405 99.1  0.2 381180 49920 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12406 99.2  0.2 381340 50064 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12407 99.2  0.2 315640 49908 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12408 99.2  0.2 315804 50040 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12409 99.1  0.2 315804 50052 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12512  0.0  0.0  63760   920 ?        R    14:08   0:00 /bin/ps u -u araim1
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
araim1   31143  0.0  0.0  63868  1084 ?        S    14:05   0:00 /bin/bash /var/spool/slurmd/job59512/slurm_script ...
araim1   31145  0.0  0.0  49388  2080 ?        S    14:05   0:00 mpirun -np 1 R --no-save
araim1   31146 75.5  1.3 592456 324488 ?       RLl  14:05   1:59 /usr/lib64/R/bin/exec/R --no-save
araim1   31153  0.0  0.0  89188  4120 ?        Sl   14:05   0:00 srun --nodes=1 --ntasks=1 --kill-on-bad-exit ...
araim1   31157 78.7  0.2 315624 49892 ?        RLl  14:05   2:04 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31158 99.2  0.2 315628 49888 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31159 59.7  0.2 315624 49900 ?        RLl  14:05   1:34 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31160 99.1  0.2 315792 50040 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31161 86.0  0.2 315628 49892 ?        RLl  14:05   2:15 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31162 99.2  0.2 360816 51252 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31163 98.2  0.2 381332 50064 ?        RLl  14:05   2:35 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31165 99.2  0.2 381668 50352 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31274  0.0  0.0  63748   912 ?        R    14:08   0:00 /bin/ps u -u araim1

This output is useful, but it would be nice to have each block of "ps" output marked with the host that it came from. Note that we can't put very complicated Bash commands into the srun command line. But we can call standalone scripts from the srun command. Consider the following script which will be run on each host.

#!/bin/bash

HOSTNAME=$(hostname)
TMP="cmd-$HOSTNAME-$$.out"

echo "----------------- $HOSTNAME -------------------" > $TMP
ps u -u $USER >> $TMP

cat $TMP
rm $TMP

Download: ../code/interacting-with-nodes/diagnostics.bash

Notice that all output is written to a temporary file, then the file is echo'ed to stdout, and then deleted. If we do not echo the output all at once, the srun command can mix up the output from different hosts making it difficult to read. We can now run our diagnostics script on each node used by our job.

[araim1@maya-usr1 ~]$ srun --jobid 59513 ./diagnostics.bash
----------------- n2 -------------------
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
araim1   12400  0.0  0.0  40748  1852 ?        S    14:05   0:00 /usr/cluster/openmpi/1.3.3-p1/gcc/4/bin/orted ...
araim1   12402 99.2  0.2 381676 50368 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12403 99.2  0.2 360664 51112 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12404 99.2  0.2 315640 49884 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12405 99.1  0.2 381180 49920 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12406 99.2  0.2 381340 50064 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12407 99.2  0.2 315640 49908 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12408 99.2  0.2 315804 50040 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12409 99.1  0.2 315804 50052 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   12512  0.0  0.0  63760   920 ?        R    14:08   0:00 /bin/ps u -u araim1
----------------- n1 -------------------
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
araim1   31143  0.0  0.0  63868  1084 ?        S    14:05   0:00 /bin/bash /var/spool/slurmd/job59512/slurm_script ...
araim1   31145  0.0  0.0  49388  2080 ?        S    14:05   0:00 mpirun -np 1 R --no-save
araim1   31146 75.5  1.3 592456 324488 ?       RLl  14:05   1:59 /usr/lib64/R/bin/exec/R --no-save
araim1   31153  0.0  0.0  89188  4120 ?        Sl   14:05   0:00 srun --nodes=1 --ntasks=1 --kill-on-bad-exit ...
araim1   31157 78.7  0.2 315624 49892 ?        RLl  14:05   2:04 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31158 99.2  0.2 315628 49888 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31159 59.7  0.2 315624 49900 ?        RLl  14:05   1:34 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31160 99.1  0.2 315792 50040 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31161 86.0  0.2 315628 49892 ?        RLl  14:05   2:15 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31162 99.2  0.2 360816 51252 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31163 98.2  0.2 381332 50064 ?        RLl  14:05   2:35 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31165 99.2  0.2 381668 50352 ?        RLl  14:05   2:36 /usr/lib64/R/bin/exec/R --slave --no-restore ...
araim1   31274  0.0  0.0  63748   912 ?        R    14:08   0:00 /bin/ps u -u araim1
[araim1@maya-usr1 ~]$