Cluster Support
Basic Commands
Access the Cluster
ssh - secure terminal service for account login
sftp/scp - file transfers
Your Account
Default shell account is bash.
echo $PATH - view current directory paths for your account
Add paths to your account temporarily: PATH=$PATH:/directory/path/to_add
Add paths to your account permanently: Edit path statement (shown below) in "~/.bash_profile". Separate paths with colons.
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATHset - useful information about path and other account settings
Cluster Status
gsh uptime - see the uptime of each compute node
gsh ps U username - see processes you have running on compute nodes
pbsnodes -a - see the current state of all the compute nodes
Compiling
To compile without the MPI libraries:
Intel Fortran F77, F90 Compiler
For Fortran 77 use: ifort my_program.f [ flags ] -o executable
For Fortran 90 use: ifort my_program.f90 [ flags ] -o executable
Intel C/C++ Compiler
For C use: icc my_program.c [ flags ] -o executable
For C++ use: icpc my_program.cpp [ flags ] -o executable
GNU Scientific Compilers
For Fortran use: f77 my_program.f
For C use: gcc my_program.c
To compile with the MPI libraries:
mpicc –c my_program.c or mpicc –c my_program my_program.c
mpif77 –c my_ program.f or mpif77 –c my_program my_program.f
mpicxx –c my_program.cxx or mpicxx –c my_program my_program.cxx
mpif90 –c my_program.f90 or mpif90 –c my_program my_program.f90
Linking (C compiler example):
mpicc –o my_program my_program.o –lm
For compiling and linking in one step:
mpicc –o my_program my_program.c
mpif77 –o my_program my_program.f
mpicxx –o my_program my_program.cxx
mpif90 –o my_program my_program.f90
Submitting Jobs
The Portable Batch System (PBS) is used to allocate nodes for your job and run it. It is the preferred method to run jobs on the compute nodes.
To use the PBS queue, you run jobs with qsub and monitor using qstat commands. The qsub command is used to submit a job to the PBS queue, and the qstat command is used to check on the status of a job in the PBS queue.
In order to run a job through PBS, you first create a PBS script that contains the commands to be executed. This script is then submitted to PBS using the qsub command.
The file pbs_mpi_cpi.sh is a sample file automatically created in your user area. Issue the command, “qsub pbs_mpi_cpi.sh” to submit the file to PBS. Note: The sample PBS script specifies the use of all 16 compute nodes. You should change the number of compute nodes to four (4) when you create your own PBS scripts. Otherwise, you may have a long wait until all nodes are available.
Commonly Used PBS Commands:
qstat - Shows status of PBS batch jobs.
Use 'qstat -a' for more detailed output, 'qstat -f' for full job details, or 'qstat -n' to see the nodes used by a particular job.qsub - Submits a job to the PBS queuing system.
qdel - Delete a PBS job from the queue.
Other PBS Commands:
qalter - Modifies the attributes of a job.
qhold - Requests that the PBS server place a hold on a job.
qmove - Remove a job from the queue in which it resides and place the job in another queue.
qmsg - Send message to PBS batch job. To send a message to a job is to write a message string into one or more output files of the job.
qorder - Exchange the order of two PBS batch jobs in a queue.
qrerun - Reruns a PBS batch job.
qrls - Release hold on PBS batch job.
qselect - List the job identifier of those jobs which meet a list of selection criteria.
qsig - Requests that a signal be send to the session leader of a batch job.
System Information
The Compute Cluster was installed on October 19, 2005 and became available to users the following day. System warranty included two years full parts and labor, three year extended manufacturers warranty and unlimited telephone and email technical support for the life of the cluster.
System Hardware
Head Node (1)
Intel 7520 Chipset Dual Processor Motherboard, 800MHz FSB Dual Core Intel EM64T 3.2GHz, 1MB cache Processors with heat sink and fan
8GB PC3200 DDR II 400 ECC REG
(2) 400GB SATA Hard Drives, 7200 RPM for OS, Mirrored Raid 1
(2) Gigabit 1000BaseT NIC
RedHat Enterprise LINUX configured for Beowulf cluster including MPICH and/or PVM
Raid Array
Ultra SCSI 320/SATA Array, provides 4TB of accessible storage
Intel 80321 RISC CPU
512MB DDR
(16) Hot Swap SATA Hard Drive Trays
(16) 300GB SATA Hard Drives, 7200 RPM
Ultra SCSI 320 Controller
Compute Nodes (16)
Intel 7520 Chipset Dual Processor Motherboard, 800MHz FSB Dual Core Intel EM64T 3.2GHz, 1MB cache Processors with heat sink and fan
8GB PC3200 DDR II 400 ECC REG
250GB UDMA Hard Drives, 7200 RPM
Gigabit 1000BaseT NIC
RedHat Enterprise LINUX configured for Beowulf cluster including MPICH and/or PVM
Other Hardware
48 Port 10/100/1000 BaseT Gigabit Switch
(5) MGE Evolution VA2200 UPS Units
System Software
Operating System
RedHat Enterprise Linux High Performance Computing License for (17) Nodes
Compilers
Intel Fortran F77, F90 Compiler, Academic Version
Intel C/C++ Compiler, Academic Version
GNU Scientific Compilers
Other System Software
LAPACK Libraries, BLAS Libraries
MPICH
OpenPBS Batch Scheduling System or Sun Grid Engine