How to run Phantom on gSTAR
Important
This machine is decommissioned.
Apply for an account
https://supercomputing.swin.edu.au/account-management/new_account_request
on Ozstar: From your account management page, request “join project” and select “oz015 - Price/Pinte research group”
on Gstar: If you are in Daniel Price’s research group, click on “Join existing project” and use the “Magnetic fields in star formation” project
First time you log in
Log in (on ozstar):
$ ssh -Y USERNAME@ozstar.swin.edu.au
or on gstar:
$ ssh -Y USERNAME@g2.hpc.swin.edu.au
show available software
$ module avail
load intel compilers
$ module load intel
get phantom
Clone a copy of phantom into your home directory
$ git clone https://github.com/danieljprice/phantom
Setting your username and email address
Ensure that your name and email address are set, as follows:
cd phantom
git config user.name "Joe Bloggs"
git config user.email "joe.bloggs@monash.edu"
Please use your full name in the format above, as this is what appears in the commit logs (and in the AUTHORS file).
get splash
Finally, install splash in your home directory by following the instructions on the splash home page
I put the “module load” commands in a file called ~/.modules which contains the modules I want loaded every time I log in. For example:
$ cat .modules
module load fftw
module load intel/13.3.1
Then, add the following lines to your ~/.bashrc
source ~/.modules
export SYSTEM=g2
ulimit -s unlimited
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HOME}/splash/giza/lib
export PATH=${PATH}:${HOME}/splash/bin
export PATH=${PATH}:/opt/torque/bin:/opt/moab/bin
export OMP_STACKSIZE=512M
Now, when you login again, all these should be set automatically.
Performing a calculation
You should not perform calculations in your home space - this is for code and small files. Calculations should be run in the “project” area in /fred/PROJECT_NAME/USERNAME
I usually make a soft link / shortcut called “runs” pointing to the directory where I want to run my calculations:
On Ozstar:
$ cd /fred/oz015
$ mkdir USERNAME
$ cd
$ ln -s /fred/oz015/USERNAME runs
$ cd runs
$ pwd -P
/fred/oz015/USERNAME
On gstar:
$ cd /lustre/projects/p035_astro
$ mkdir USERNAME
$ cd
$ ln -s /lustre/projects/p035_astro/USERNAME runs
$ cd runs
$ pwd -P
/lustre/projects/p035_astro/USERNAME
then make a subdirectory for the name of the calculation you want to run (e.g. shock)
$ mkdir shock
$ cd shock
$ ~/phantom/scripts/writemake.sh shock > Makefile
$ make setup
$ make
$ ./phantomsetup myshock
To run the code, you need to write a .pbs script. You can get an example by typing “make qscript”:
$ make qscript INFILE=myshock.in > run.pbs
should produce something like
$ cat run.pbs
#!/bin/bash
## PBS Job Submission Script, created by "make qscript" Mon Mar 6 09:34:55 AEDT 2017
#PBS -l nodes=1:ppn=16
#PBS -N preconference
#PBS -q sstar
#PBS -o myshock.in.pbsout
#PBS -j oe
#PBS -m e
#PBS -M daniel.price@monash.edu
#PBS -l walltime=168:00:00
#PBS -l mem=16G
## phantom jobs can be restarted:
#PBS -r y
cd $PBS_O_WORKDIR
echo "PBS_O_WORKDIR is $PBS_O_WORKDIR"
echo "PBS_JOBNAME is $PBS_JOBNAME"
env | grep PBS
cat $PBS_NODEFILE > nodefile
echo "HOSTNAME = $HOSTNAME"
echo "HOSTTYPE = $HOSTTYPE"
echo Time is `date`
echo Directory is `pwd`
ulimit -s unlimited
export OMP_SCHEDULE="dynamic"
export OMP_NUM_THREADS=16
echo "starting phantom run..."
export outfile=`grep logfile "myshock.in" | sed "s/logfile =//g" | sed "s/\\!.*//g" | sed "s/\s//g"`
echo "writing output to $outfile"
./phantom myshock.in >& $outfile
You can then submit this to the queue using
$ qsub run.pbs
5134395.pbs.hpc.swin.edu.au
and check status using
$ qstat
5134395.pbs preconference dprice 0 Q sstar
check when your job is definitely going to start by using
$ showstart
more info
For more information on the actual machine read the userguide