Tips and tricks for running on Pawsey machines
==============================================

Magnus
------

Use:

::

   export SYSTEM=cray

for the Cray Fortran compiler

When compiling on the head node, sometimes you get

::

   /bin/bash: line 1: 16605 Illegal instruction     ../bin/phantom test

which is caused by different instruction sets (Haswell vs. Sandy Bridge)

::

   module swap craype-haswell craype-sandybridge

Apparently magnus-1 and magnus-2 are Sandybridge while magnus-3 and
magnus-4 are Haswell.

Allinea MAP profiling tools
---------------------------

::

   module load forge

you need to first create the libraries needed to run the map tool:

::

   cd $HOME
   mkdir allinea
   cd allinea
   make-profiler-libraries

then phantom needs to be compiled with some extra flags in
build/Makefile:

::

   FFLAGS+=-G2
   LDFLAGS+= -L$HOME/allinea -lmap ... < and some other stuff >

then to run Allinea on openMP code in the run script need:

::

   export ALLINEA_NO_MPI_AUTODETECT=1
   map --profile aprun -n 1 -d 24 ./phantom disc.in

CrayPAT profiling tools
-----------------------

::

   module load perftools/6.1.0

First compile Phantom using SYSTEM=cray (i.e. using CRAY Fortran
compiler):

::

   make SYSTEM=cray

Then use:

::

   pat_build -o phantom-pat

then submit to queue using the execute command

::

   <rest of job script>
   aprun -n 1 -d 24 ./phantom+pat disc.in

then should output phantom+pat-.xf. Then run

::

   pat_report <name of .xf file> > sampling.txt

This produces sampling.txt, also .apa and .ap2 files. Latter is for
viewing using Apprentice2:

::

   app2 phantom*.ap2

Typical job script for scaling and profiling tests
--------------------------------------------------

::

   #!/bin/bash --login
   #SBATCH --account=director2006
   #SBATCH --nodes=1
   #SBATCH --job-name=scaling
   #SBATCH --output=disc.in.pbsout
   #SBATCH --mail-type=FAIL
   #SBATCH --mail-type=END
   #SBATCH --mail-user=daniel.price@monash.edu
   #SBATCH --time=03:00:00
   #SBATCH --mem=10000M

   #module load intel
   #module load mpt
   #export SYSTEM=ifort

   ulimit -s unlimited
   export OMP_SCHEDULE="dynamic"
   ##export OMP_NUM_THREADS=24
   export OMP_STACKSIZE=1024M
   export LD_LIBRARY_PATH=/scratch/director2006/rnealon/runs/replicate_magnus/allinea:$LD_LIBRARY_PATH
   export ALLINEA_NO_MPI_AUTODETECT=1

   echo "starting phantom run..."
   export outfile=`grep logfile "disc.in" | sed "s/logfile =//g" | sed "s/\\!.*//g" | sed "s/\s//g"`
   echo "writing output to $outfile"

   for x in 1 2 4 12 24; do
      export OMP_NUM_THREADS=$x
      map --profile aprun -n 1 -d $OMP_NUM_THREADS ./phantom blast.in >& $outfile
      cp blast01.log $x\_thread.log
      cp blast.in.s blast.in
   done

Using intel instead of cray Fortran compiler
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

   module swap PrgEnv-cray PrgEnv-intel

Also PrgEnv-gnu for Gnu compilers. This makes ftn->ifort and cc->icc.
Also links various libraries appropriate to the compiler. MPI code also
compiles with just ftn and cc instead of mpif90 and mpicc.

Getting an interactive node
~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

   salloc -p debugq