Getting started on Kennedy (local cluster at the University of St Andrews)
Apply for an account
Talk to your line manager about getting an account.
First time you log in
Log in with your username:
$ ssh -Y USERNAME@kennedy.st-andrews.ac.uk
Load intel compiler (currently Phantom works best with this compiler)
$ source /opt/intel/oneapi/setvars.sh >> /dev/null 2>&1
Add the above command and the following three commands to your ~/.bashrc:
export SYSTEM=kennedy ulimit -s unlimited export OMP_STACKSIZE=512M
Now, when you login again, all these should be set automatically. To load these settings for this current session, use
This command is only required when you modify your bashrc.
Clone a copy of Phantom into your HOME directory:
$ cd ~ $ git clone https://github.com/danieljprice/phantom
Your home directory has a 10Gb quota. Do NOT run simulations in your home directory.
Performing a calculation
All calculations must be performed in your SCRATCH directory. Make a directory (within your user folder) for runs:
$ cd /gpfs1/scratch/astro/USERNAME $ mkdir runs $ cd runs
Next, make a subdirectory for the name of the calculation you want to run (e.g. shock)
$ mkdir shock $ cd shock $ ~/phantom/scripts/writemake.sh shock > Makefile $ make ; make setup
Run phantomsetup twice to create your initial conditions:
$ ./phantomsetup shock (just press enter to all the questions to get the default) $ ./phantomsetup shock
The first instance of phantomsetup will generate shock.setup. The second instance of phantomsetup will read shock.setup and generate the initial dump file, shock_00000.tmp, and input file, shock.in.
An example submission script, run.qscript should look something like
#!/bin/bash -l #SBATCH -J shock #SBATCH --nodes=1 #SBATCH -p singlenode #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=32 #SBATCH --time=168:00:00 #SBATCH --output=shock_output.%j #SBATCH --error=shock_error.%j #SBATCH --mail-type=BEGIN,END,FAIL #SBATCH --mail-user=[enter your email here] export OMP_NUM_THREADS=32 export KMP_STACKSIZE=128M ulimit -s unlimited echo $SLURM_NPROCS date cd [current working directory] ./phantom shock.in > shock01.log date
Note that there are differences between the above script and that generated by Phantom’s make qscript. You will need to enter the email destination, job name (after -J) and path (without the brackets).
On Kennedy, the –time entry is optional. If this entry is excluded, then the default wall time of 30 days (=720hours) will be used. Likewise, you can manually select any wall time between 0-720h.
For short jobs or for testing, you can submit your script to the development queue. Do this by resetting the wall time to a maximum of two hours and selecting the queue:
#SBATCH --time==2:00:00 #SBATCH -p debug
You can then submit this to the queue using
$ sbatch run.qscript Submitted batch job 375869
and check status using the squeue command and your username, e.g.
$ squeue -u [your username] JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 375869 singlenod shock jhw PD 0:00 1 (Priority) 375527 singlenod shockA jhw R 1-13:34:53 1 kennedy119
If your simulation has not yet started, you can see when it is predicted to start by
$ squeue --start -u [USERNAME]
You can cancel a run (before or during execution) by
$ scancel [Job ID]
When the job has started, you can follow what the calculation is doing by looking at the .log file:
$ tail -f shock01.log
(press ctrl-c to quit the tail -f command). You should obtain a series of dump files:
$ ls shock_00000 shock_00001 shock_00002
You can check conserved quantities by plotting things in the .ev file. The first line of the file shows you what each column is:
$ head shock01.ev
and you can plot these columns using your favourite graphing programme.
When running your own simulation, use the name of the relevant setup block when making the Makefile:
$ ~/phantom/scripts/writemake.sh [setup block name] > Makefile
The setup blocks are listed in /build/Makefile_setups. The model name can be anything you choose; in the above example, the model name is ‘shock’. Naturally, the name you choose will replace all instances of ‘shock’ above (except when generating the local Makefile).
Finally, install splash in your home directory by following the instructions on the splash home page
More info is available on the website: https://www.st-andrews.ac.uk/high-performance-computing/