Getting started on Kennedy (local cluster at the University of St Andrews)
==============================================================================

Apply for an account
--------------------

Talk to your line manager about getting an account.

First time you log in
---------------------

Log in with your username:

::

   $ ssh -Y USERNAME@kennedy.st-andrews.ac.uk


Load intel compiler (currently Phantom works best with this compiler)

::

   $ source /opt/intel/oneapi/setvars.sh >> /dev/null 2>&1

Add the above command and the following three commands to your ~/.bashrc:

::

   export SYSTEM=kennedy
   ulimit -s unlimited
   export OMP_STACKSIZE=512M

Now, when you login again, all these should be set automatically.  To load these settings for this current session, use

::

   source ~/.bashrc

This command is only required when you modify your bashrc.

Get Phantom
~~~~~~~~~~~

Clone a copy of Phantom into your HOME directory:

::

   $ cd ~
   $ git clone https://github.com/danieljprice/phantom

Your home directory has a 10Gb quota.  Do NOT run simulations in your home directory.

Performing a calculation
------------------------

All calculations must be performed in your SCRATCH directory.  Make a directory (within your user folder) for runs:

::

   $ cd /gpfs1/scratch/astro/USERNAME
   $ mkdir runs
   $ cd runs

Next, make a subdirectory for the name of the calculation you want to run (e.g. shock)

::

   $ mkdir shock
   $ cd shock
   $ ~/phantom/scripts/writemake.sh shock > Makefile
   $ make ; make setup

Run phantomsetup twice to create your initial conditions:

::

   $ ./phantomsetup shock
   (just press enter to all the questions to get the default)
   $ ./phantomsetup shock

The first instance of phantomsetup will generate shock.setup.  The second instance of phantomsetup will read shock.setup and generate the initial dump file, shock_00000.tmp, and input file, shock.in.

An example submission script, run.qscript should look something like

::

   #!/bin/bash -l
   #SBATCH -J shock
   #SBATCH --nodes=1
   #SBATCH -p singlenode 
   #SBATCH --ntasks-per-node=1 
   #SBATCH --cpus-per-task=32 
   #SBATCH --time=168:00:00  
   #SBATCH --output=shock_output.%j
   #SBATCH --error=shock_error.%j
   #SBATCH --mail-type=BEGIN,END,FAIL 
   #SBATCH --mail-user=[enter your email here]

   export OMP_NUM_THREADS=32 
   export KMP_STACKSIZE=128M 
   ulimit -s unlimited 

   echo $SLURM_NPROCS 
   date 
   cd [current working directory]
   ./phantom shock.in > shock01.log
   date 

Note that there are differences between the above script and that generated by Phantom's make qscript. You will need to enter the email destination, job name (after -J) and path  (without the brackets).

On Kennedy, the --time entry is optional.  If this entry is excluded, then the default wall time of 30 days (=720hours) will be used.  Likewise, you can manually select any wall time between 0-720h.  

For short jobs or for testing, you can submit your script to the development queue.  Do this by resetting the wall time to a maximum of two hours and selecting the queue:

::

   #SBATCH --time==2:00:00
   #SBATCH -p debug 

You can then submit this to the queue using

::

   $ sbatch run.qscript
   Submitted batch job 375869

and check status using the squeue command and your username, e.g.

::

   $ squeue -u [your username]
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            375869 singlenod    shock     jhw  PD       0:00      1 (Priority)
            375527 singlenod   shockA     jhw   R 1-13:34:53      1 kennedy119

If your simulation has not yet started, you can see when it is predicted to start by

::

   $ squeue --start -u [USERNAME]

You can cancel a run (before or during execution) by

::

   $ scancel [Job ID]

When the job has started, you can follow what the calculation is doing by
looking at the .log file:

::

   $ tail -f shock01.log

(press ctrl-c to quit the tail -f command). You should obtain a series
of dump files:

::

   $ ls
   shock_00000
   shock_00001
   shock_00002

You can check conserved quantities by plotting things in the .ev
file. The first line of the file shows you what each column is:

::

   $ head shock01.ev

and you can plot these columns using your favourite graphing programme.


Model names
-----------

When running your own simulation, use the name of the relevant setup block when making the Makefile:

::

   $ ~/phantom/scripts/writemake.sh [setup block name] > Makefile

The setup blocks are listed in /build/Makefile_setups.  The model name can be anything you choose; in the above example, the model name is 'shock'.  Naturally, the name you choose will replace all instances of 'shock' above (except when generating the local Makefile).


get splash
~~~~~~~~~~

Finally, install splash in your home directory by following the
instructions on the `splash home
page <http://users.monash.edu.au/~dprice/splash/>`__

More info
---------

More info is available on the website:
https://www.st-andrews.ac.uk/high-performance-computing/