Roofline models

Roofline on Cascade-Lake CPU (Darwin) with Intel-tools

Step 1: Compile with Intel

This worked for me (on a cascade-lake node):

module load gcc/9.3.0 intel/20.0.1
source /projects/parthenon-int/parthenon-project/.bashrc
cmake -S. -DCMAKE_CXX_COMPILER=icpc -DCMAKE_C_COMPILER=icc -Bbuildcsdlk
cmake --build buildcsdlk

Step 2: Collect roofline data

Note: In very new versions of the Intel-Advisor (~2020-ish?), this can be done in one step with -collect roofline [-flop]. In older versions, this is a 2-part process.

First, you need to load the model that has Intel-Advisor in it. On Darwin, this is intel-performance-tools/18.0.3 (that seems to be the newest available there):

module load intel-performance-tools/18.0.3

For single-MPI rank/serial runs:

Part 1:

srun -n 1 --cpu_bind=cores advixe-cl -collect survey --project-dir ./my_rl -search-dir all:r=$HOME/parthnstuff/parth2/buildcsdlk/ -- $HOME/parthnstuff/parth2/buildcsdlk/example/advection/advection-example -i ./parthinput.advection

Part 2:

srun -n 1 --cpu_bind=cores advixe-cl -collect tripcounts -flop [-stacks] --project-dir ./my_rl -search-dir all:r=$HOME/parthnstuff/parth2/buildcsdlk/ -- $HOME/parthnstuff/parth2/buildcsdlk/example/advection/advection-example -i ./parthinput.advection

N.b.: add the -stacks option if you want to collect the callstacks as well.

For multi-MPI rank runs:

Create a run script that runs the above advixe-cl- commands on only one or two ranks (and writes to separate directories; I don't know if this is strictly necessary, but it worked):

run_survey.sh:

#!/bin/bash
if [ $SLURM_PROCID -eq 0 ]
then
    advixe-cl -collect survey --project-dir ./my_rl_0 -search-dir all:r=$HOME/codes/parthnstuff/parth2/buildcsdlk/ -- $HOME/codes/parthnstuff/parth2/buildcsd2/example/advection/advection-example -i ./parthinput.advection
elif [ $SLURM_PROCID -eq 1 ]
then
    advixe-cl -collect survey --project-dir ./my_rl_1 -search-dir all:r=$HOME/codes/parthnstuff/parth2/buildcsdlk/ -- $HOME/codes/parthnstuff/parth2/buildcsd2/example/advection/advection-example -i ./parthinput.advection
else
    $HOME/codes/parthnstuff/parth2/buildcsd2/example/advection/advection-example -i ./parthinput.advection
fi

$ srun -n 2 run_survey.sh

and similarly for the -collect tripcounts option.

Step 3: Analyse Data In GUI:

Make sure you have a viable way of opening/interacting with X11 windows, then open the GUI:

advixe-gui

Choose Open Results, navigate to the 'project-dir' where you collected the roofline data (and possibly another level or two deeper), and select the *.advixeexp in there.

To view the roofline graph, select the tab "Survey & Roofline" (near the middle-left at the top), and find a vertical bar labeled "Roofline". Click on that.

If you collected data on multiple MPI ranks, each roofline model has to be opened separately.