UT Arlington Physics Department

Cluster Support

Basic Commands

Access the Cluster

ssh - secure terminal service for account login

sftp/scp - file transfers

Your Account

Default shell account is bash.

echo $PATH - view current directory paths for your account

Add paths to your account temporarily: PATH=$PATH:/directory/path/to_add

Add paths to your account permanently: Edit path statement (shown below) in "~/.bash_profile". Separate paths with colons.

# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH

set - useful information about path and other account settings

Cluster Status

gsh uptime - see the uptime of each compute node

gsh ps U username - see processes you have running on compute nodes

pbsnodes -a - see the current state of all the compute nodes

Compiling

To compile without the MPI libraries:

Intel Fortran F77, F90 Compiler

For Fortran 77 use: ifort my_program.f [ flags ] -o executable

For Fortran 90 use: ifort my_program.f90 [ flags ] -o executable

Intel C/C++ Compiler

For C use: icc my_program.c [ flags ] -o executable

For C++ use: icpc my_program.cpp [ flags ] -o executable

GNU Scientific Compilers

For Fortran use: f77 my_program.f

For C use: gcc my_program.c

To compile with the MPI libraries:

mpicc –c my_program.c   or   mpicc –c my_program my_program.c

mpif77 –c my_ program.f   or   mpif77 –c my_program my_program.f

mpicxx –c my_program.cxx   or   mpicxx –c my_program my_program.cxx

mpif90 –c my_program.f90   or   mpif90 –c my_program my_program.f90

Linking (C compiler example):

mpicc –o my_program my_program.o –lm

For compiling and linking in one step:

mpicc –o my_program my_program.c

mpif77 –o my_program my_program.f

mpicxx –o my_program my_program.cxx

mpif90 –o my_program my_program.f90

Submitting Jobs

The Portable Batch System (PBS) is used to allocate nodes for your job and run it. It is the preferred method to run jobs on the compute nodes.

To use the PBS queue, you run jobs with qsub and monitor using qstat commands. The qsub command is used to submit a job to the PBS queue, and the qstat command is used to check on the status of a job in the PBS queue.
In order to run a job through PBS, you first create a PBS script that contains the commands to be executed. This script is then submitted to PBS using the qsub command.

The file pbs_mpi_cpi.sh is a sample file automatically created in your user area. Issue the command, “qsub pbs_mpi_cpi.sh” to submit the file to PBS. Note: The sample PBS script specifies the use of all 16 compute nodes. You should change the number of compute nodes to four (4) when you create your own PBS scripts. Otherwise, you may have a long wait until all nodes are available.

Commonly Used PBS Commands:

qstat - Shows status of PBS batch jobs.
Use 'qstat -a' for more detailed output, 'qstat -f' for full job details, or 'qstat -n' to see the nodes used by a particular job.

qsub - Submits a job to the PBS queuing system.

qdel - Delete a PBS job from the queue.

Other PBS Commands:

qalter - Modifies the attributes of a job.

qhold - Requests that the PBS server place a hold on a job.

qmove - Remove a job from the queue in which it resides and place the job in another queue.

qmsg - Send message to PBS batch job. To send a message to a job is to write a message string into one or more output files of the job.

qorder - Exchange the order of two PBS batch jobs in a queue.

qrerun - Reruns a PBS batch job.

qrls - Release hold on PBS batch job.

qselect - List the job identifier of those jobs which meet a list of selection criteria.

qsig - Requests that a signal be send to the session leader of a batch job.