Tips and tricks for running on Pawsey machines
Magnus
Use:
export SYSTEM=cray
for the Cray Fortran compiler
When compiling on the head node, sometimes you get
/bin/bash: line 1: 16605 Illegal instruction ../bin/phantom test
which is caused by different instruction sets (Haswell vs. Sandy Bridge)
module swap craype-haswell craype-sandybridge
Apparently magnus-1 and magnus-2 are Sandybridge while magnus-3 and magnus-4 are Haswell.
Allinea MAP profiling tools
module load forge
you need to first create the libraries needed to run the map tool:
cd $HOME
mkdir allinea
cd allinea
make-profiler-libraries
then phantom needs to be compiled with some extra flags in build/Makefile:
FFLAGS+=-G2
LDFLAGS+= -L$HOME/allinea -lmap ... < and some other stuff >
then to run Allinea on openMP code in the run script need:
export ALLINEA_NO_MPI_AUTODETECT=1
map --profile aprun -n 1 -d 24 ./phantom disc.in
CrayPAT profiling tools
module load perftools/6.1.0
First compile Phantom using SYSTEM=cray (i.e. using CRAY Fortran compiler):
make SYSTEM=cray
Then use:
pat_build -o phantom-pat
then submit to queue using the execute command
<rest of job script>
aprun -n 1 -d 24 ./phantom+pat disc.in
then should output phantom+pat-.xf. Then run
pat_report <name of .xf file> > sampling.txt
This produces sampling.txt, also .apa and .ap2 files. Latter is for viewing using Apprentice2:
app2 phantom*.ap2
Typical job script for scaling and profiling tests
#!/bin/bash --login
#SBATCH --account=director2006
#SBATCH --nodes=1
#SBATCH --job-name=scaling
#SBATCH --output=disc.in.pbsout
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=END
#SBATCH --mail-user=daniel.price@monash.edu
#SBATCH --time=03:00:00
#SBATCH --mem=10000M
#module load intel
#module load mpt
#export SYSTEM=ifort
ulimit -s unlimited
export OMP_SCHEDULE="dynamic"
##export OMP_NUM_THREADS=24
export OMP_STACKSIZE=1024M
export LD_LIBRARY_PATH=/scratch/director2006/rnealon/runs/replicate_magnus/allinea:$LD_LIBRARY_PATH
export ALLINEA_NO_MPI_AUTODETECT=1
echo "starting phantom run..."
export outfile=`grep logfile "disc.in" | sed "s/logfile =//g" | sed "s/\\!.*//g" | sed "s/\s//g"`
echo "writing output to $outfile"
for x in 1 2 4 12 24; do
export OMP_NUM_THREADS=$x
map --profile aprun -n 1 -d $OMP_NUM_THREADS ./phantom blast.in >& $outfile
cp blast01.log $x\_thread.log
cp blast.in.s blast.in
done
Using intel instead of cray Fortran compiler
module swap PrgEnv-cray PrgEnv-intel
Also PrgEnv-gnu for Gnu compilers. This makes ftn->ifort and cc->icc. Also links various libraries appropriate to the compiler. MPI code also compiles with just ftn and cc instead of mpif90 and mpicc.
Getting an interactive node
salloc -p debugq