Getting started on Rusty (Flatiron cluster)

We assume you already have a Flatiron username and account

First time you log in

Make sure you log in with the -Y flag to enable X-Windows forwarding:

ssh -Y -p 61022 <USER>@gateway.flatironinstitute.org
ssh -Y rusty

show available software:

module avail

load intel compilers and splash

module load intel-oneapi-compilers/2023.0.0
module use ~yjiang/modulefiles
module load splash/3.8.3

get phantom

Clone a copy of phantom into your home directory:

git clone https://github.com/danieljprice/phantom.git

set your username and email address

Ensure that your name and email address are set, as follows:

cd phantom
git config --global user.name "Joe Bloggs"
git config --global user.email "joe.bloggs@monash.edu"

Please use your full name in the format above, as this is what appears in the commit logs (and in the AUTHORS file).

edit your .bashrc file

I put the “module load” commands in a file called ~/.modules which contains the modules I want every time I log in. For example:

$ cat .modules
module load intel-oneapi-compilers/2023.0.0

Then, add the following lines to your ~/.bashrc:

source ~/.modules
export SYSTEM=rusty
ulimit -s unlimited
export OMP_STACKSIZE=512M
export OMP_SCHEDULE=dynamic

Now, when you login again, all these should be set automatically.

Performing a calculation

You should not perform calculations in your home space - this is for code and small files. Calculations should be run in the “ceph” area in /mnt/ceph/users/$USER/

On other machines I usually make a soft link / shortcut called “runs” pointing to the directory where I want to run my calculations:

$ cd
$ ln -s /mnt/ceph/users/$USER runs
$ cd runs
$ pwd -P
/mnt/ceph/users/USERNAME

However on the Flatiron machines there is already a shortcut called “ceph” in your homespace.

Then make a subdirectory for the name of the calculation you want to run (e.g. shock):

mkdir shock
cd shock
~/phantom/scripts/writemake.sh shock > Makefile
make setup
make
./phantomsetup shock

To run the code, you need to write a slurm script. You can get an example by typing “make qscript”:

make qscript NOMP=10 INFILE=shock.in > run.q

should produce something like:

$ cat run.q
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=10
#SBATCH --job-name=audiencia
#SBATCH --partition=gen
#SBATCH --output=shock.in.qout
#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=END
#SBATCH --mail-user=daniel.price@monash.edu
#SBATCH --time=0-168:00:00
#SBATCH --mem=16G
echo "HOSTNAME = $HOSTNAME"
echo "HOSTTYPE = $HOSTTYPE"
echo Time is `date`
echo Directory is `pwd`

ulimit -s unlimited
export OMP_SCHEDULE="dynamic"
export OMP_NUM_THREADS=10
export OMP_STACKSIZE=1024m

echo "starting phantom run..."
export outfile=`grep logfile "shock.in" | sed "s/logfile =//g" | sed "s/   \\!.*//g" | sed "s/\s//g"`
echo "writing output to $outfile"
./phantom shock.in >& $outfile

You can then submit this to the “temp” queue using:

$ sbatch -p temp run.q
Submitted batch job 2547013

check status using:

$ squeue -u $USER
          JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
        2547013      temp audienci   dprice  R       0:02      1 worker3109

Once the job is running, follow the output log using the tail -f command:

tail -f shock01.log

splash on rusty

There is a version of splash you can get by loading the relevant module:

module use ~yjiang/modulefiles
module load splash/3.8.3

Alternatively, you can also install a local copy of splash in your home space:

cd
git clone https://github.com/danieljprice/splash
cd splash; git clone https://github.com/danieljprice/giza
make withgiza SYSTEM=ifort

You can add this directory in your path by putting the following lines in your ~/.bashrc file:

export SPLASH_DIR=/mnt/home/dprice/splash
export PATH=${SPLASH_DIR}/bin:${PATH}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${SPLASH_DIR}/giza/lib

visualising the shock problem

Simple plots with splash can be launched from the command line, e.g.:

splash shock_0* -y density -x x

then press space to step through the snapshots. You can plot the exact solution for the shock tube by switching it on from the o) menu:

Please enter your selection now (y axis or option):o
------------- particle plot options -------------------
8) plot exact solution                 (  3 )
...
enter option ([0:9], default=0): 8
...
3) shock tube
...
enter exact solution to plot ([0:18], default=3):
plotting exact solution number  3
no file sod.shk
 >> read sod.setup
 rhoL, rho_R =    1.0000000000000000       0.12500000000000000
 pr_L, pr_R  =    1.0000000000000000       0.10000000000000001
 v_L,  v_R   =    0.0000000000000000        0.0000000000000000
set adiabatic gamma manually? (no=read from dumps) (default=no):

then press ‘s’ from the main menu to save, and remake your plot:

Please enter your selection now (y axis or option):6
(x axis) (default=1):
(render) (0=none) ([0:16], default=0):
Graphics device/type (? to see list, default /xw):

To make an offline version of the same plot from the command line, use:

splash shock_0* -y density -x x -dev shock.pdf

read the splash userguide for more

more interesting examples

To proceed to a more interesting calculation, just change the name of the SETUP parameter when you created the Makefile in the run directory, as per the examples:

cd ~/ceph
mkdir disc
cd disc
~/phantom/scripts/writemake.sh disc > Makefile
make setup; make

followed by:

./phantomsetup disc

after editing the .setup file as desired, then finish the setup and run the calculation:

./phantomsetup disc
make qscript NOMP=10 INFILE=disc.in > run.q
sbatch -p temp run.q

…and you’re off and rolling.

more info

For more information on the actual machine read the userguide