Getting started on Monarch (Monash campus cluster)
Apply for an account
Follow the instructions for how to apply for an account: https://docs.monarch.erc.monash.edu/MonARCH/requesting-an-account.html
If you are in Daniel Price’s research group, click on “Join existing project” and use the “pMona0001” project
First time you log in
Log in:
$ ssh -Y USERNAME@monarch.erc.monash.edu
show available software
$ module avail
load intel compilers and recent git
$ module load intel
$ module load git/2.19.0
get phantom
Clone a copy of phantom into your home directory
$ git clone https://github.com/danieljprice/phantom.git
get splash
Finally, install splash in your home directory by following the instructions on the splash home page
I put the “module load” commands in a file called ~/.modules which contains the modules I want loaded every time I log in. For example:
$ cat .modules
module load intel/2018u3
module load gcc/5.4.0
module load git/2.19.0
Then, add the following lines to your ~/.bashrc
source ~/.modules
export SYSTEM=monarch
ulimit -s unlimited
export OMP_STACKSIZE=512M
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HOME}/splash/giza/lib
export PATH=${PATH}:${HOME}/splash/bin
Now, when you login again, all these should be set automatically.
Performing a calculation
I usually make a soft link / shortcut called “runs” pointing to the directory where I want to run my calculations:
$ ln -s pMona0001/joebloggs/runs runs
$ cd runs
$ pwd -P
/mnt/lustre/projects/pMona0001/dprice/runs
then make a subdirectory for the name of the calculation you want to run (e.g. cluster)
$ mkdir shock
$ cd shock
$ ~/phantom/scripts/writemake.sh shock > Makefile
$ make setup
$ make
then run phantomsetup to create your initial conditions
$ ./phantomsetup shock
(just press enter to all the questions to get the default)
To run the code, you need to write a slurm script. You can get an example by typing “make qscript”:
$ make qscript INFILE=shock.in > run.qscript
should produce something like
$ cat run.qscript
$ make qscript INFILE=shock.in
#!/bin/bash
#SBATCH --nodes=1 --ntasks=16
#SBATCH --cpus-per-task=1
#SBATCH --job-name=buck-jump
#SBATCH --account=p01
#SBATCH --output=shock.in.qout
#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=END
#SBATCH --mail-user=daniel.price@monash.edu
#SBATCH --time=0-100:59:59
#SBATCH --mem=16G
echo "HOSTNAME = $HOSTNAME"
echo "HOSTTYPE = $HOSTTYPE"
echo Time is `date`
echo Directory is `pwd`
ulimit -s unlimited
export OMP_SCHEDULE="dynamic"
export OMP_NUM_THREADS=16
export OMP_STACKSIZE=1024m
echo "starting phantom run..."
export outfile=`grep logfile "shock.in" | sed "s/logfile =//g" | sed "s/\\!.*//g" | sed "s/\s//g"`
echo "writing output to $outfile"
./phantom shock.in >& $outfile
You can then submit this to the queue using
$ sbatch run.qscript
Submitted batch job 2162704
and check status using
$ squeue
2162702 medium extended price R 2-12:13:46 1 hs9
2162703 medium extended price R 2-12:13:46 1 hs9
2162704 medium extended price R 2-12:13:46 1 hs9
You can follow what the calculation is doing by looking at the .log file:
$ tail -f shock01.log
(press ctrl-c to quit the tail -f command). You should obtain a series of dump files:
$ ls
shock_00000
shock_00001
shock_00002
which you can view with splash (use the “ssplash” binary to view this format):
$ ssplash shock_0*
You can also check conserved quantities by plotting things in the .ev file. The first line of the file shows you what each column is:
$ head shock01.ev
and you can plot these columns using “asplash -ev” or any other program for plotting ascii files, like gnu plot:
$ asplash -ev *.ev