Getting started on Kennedy (local cluster at the University of St Andrews)

Apply for an account

Talk to your line manager about getting an account.

First time you log in

Log in with your username:

$ ssh -Y USERNAME@kennedy.st-andrews.ac.uk

Load intel compiler (currently Phantom works best with this compiler)

$ source /opt/intel/oneapi/setvars.sh >> /dev/null 2>&1

Add the above command and the following three commands to your ~/.bashrc:

export SYSTEM=kennedy
ulimit -s unlimited
export OMP_STACKSIZE=512M

Now, when you login again, all these should be set automatically. To load these settings for this current session, use

source ~/.bashrc

This command is only required when you modify your bashrc.

Get Phantom

Clone a copy of Phantom into your HOME directory:

$ cd ~
$ git clone https://github.com/danieljprice/phantom

Your home directory has a 10Gb quota. Do NOT run simulations in your home directory.

Performing a calculation

All calculations must be performed in your SCRATCH directory. Make a directory (within your user folder) for runs:

$ cd /gpfs1/scratch/astro/USERNAME
$ mkdir runs
$ cd runs

Next, make a subdirectory for the name of the calculation you want to run (e.g. shock)

$ mkdir shock
$ cd shock
$ ~/phantom/scripts/writemake.sh shock > Makefile
$ make ; make setup

Run phantomsetup twice to create your initial conditions:

$ ./phantomsetup shock
(just press enter to all the questions to get the default)
$ ./phantomsetup shock

The first instance of phantomsetup will generate shock.setup. The second instance of phantomsetup will read shock.setup and generate the initial dump file, shock_00000.tmp, and input file, shock.in.

An example submission script, run.qscript should look something like

#!/bin/bash -l
#SBATCH -J shock
#SBATCH --nodes=1
#SBATCH -p singlenode
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --time=168:00:00
#SBATCH --output=shock_output.%j
#SBATCH --error=shock_error.%j
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=[enter your email here]

export OMP_NUM_THREADS=32
export KMP_STACKSIZE=128M
ulimit -s unlimited

echo $SLURM_NPROCS
date
cd [current working directory]
./phantom shock.in > shock01.log
date

Note that there are differences between the above script and that generated by Phantom’s make qscript. You will need to enter the email destination, job name (after -J) and path (without the brackets).

On Kennedy, the –time entry is optional. If this entry is excluded, then the default wall time of 30 days (=720hours) will be used. Likewise, you can manually select any wall time between 0-720h.

For short jobs or for testing, you can submit your script to the development queue. Do this by resetting the wall time to a maximum of two hours and selecting the queue:

#SBATCH --time==2:00:00
#SBATCH -p debug

You can then submit this to the queue using

$ sbatch run.qscript
Submitted batch job 375869

and check status using the squeue command and your username, e.g.

$ squeue -u [your username]
          JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
         375869 singlenod    shock     jhw  PD       0:00      1 (Priority)
         375527 singlenod   shockA     jhw   R 1-13:34:53      1 kennedy119

If your simulation has not yet started, you can see when it is predicted to start by

$ squeue --start -u [USERNAME]

You can cancel a run (before or during execution) by

$ scancel [Job ID]

When the job has started, you can follow what the calculation is doing by looking at the .log file:

$ tail -f shock01.log

(press ctrl-c to quit the tail -f command). You should obtain a series of dump files:

$ ls
shock_00000
shock_00001
shock_00002

You can check conserved quantities by plotting things in the .ev file. The first line of the file shows you what each column is:

$ head shock01.ev

and you can plot these columns using your favourite graphing programme.

Model names

When running your own simulation, use the name of the relevant setup block when making the Makefile:

$ ~/phantom/scripts/writemake.sh [setup block name] > Makefile

The setup blocks are listed in /build/Makefile_setups. The model name can be anything you choose; in the above example, the model name is ‘shock’. Naturally, the name you choose will replace all instances of ‘shock’ above (except when generating the local Makefile).

get splash

Finally, install splash in your home directory by following the instructions on the splash home page

More info

More info is available on the website: https://www.st-andrews.ac.uk/high-performance-computing/