GPU usage of Ascent

[yslan]

Hi

1. When we specify runtine/vtkm/backend what happen if the backend is not availiable? Will it choses the CPU / OpenMP instead?

   I know we can print out the about() or the runtime actions but I'm not sure those info are indeed used since we print it outside Ascent's API.

   Or, maybe there is an API function to confirm which backend is used?

2. What GPU stream does Ascent use? Is there a already a synchronization in Ascent's API?

Thanks!

[nicolemarsaglia]

@yslan I did not test this, but most likely, if the specified backend is not available it will break expecting to be built with the correct flags and TPLs. Otherwise it should default to serial on the CPU.

Yes, another way to get information from Ascent using the API is either of the info() calls. This will fill or produce a conduit node with information that you can print out:

ascent/src/libs/ascent/ascent_runtime.hpp

Line 38 in 950c5f1

virtual void Info(conduit::Node &info_out)=0;

ascent/src/libs/ascent/ascent_runtime.hpp

Line 40 in 950c5f1

virtual conduit::Node &Info()=0;

In terms of what GPUs Ascent will work on, it is capable of using VTK-m with CUDA, and VTK-m with Kokkos and HIP. I'm currently working on an implementation using VTK-m with Kokkos and CUDA.

Let me know if you have any more questions!

[yslan]

@nicolemarsaglia Thanks for the response.
I'm actually wondering about the cudaStream_t that could cover kernel launch time and memcopy.
https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf

[nicolemarsaglia]

@yslan ah, now I see, interesting! Thanks for correcting me, I'd never heard of GPU streams before. I'm not sure of the answer. I don't think Ascent does any GPU streaming or synchronization directly, that is handled by either VTKm or Kokkos, and even then I'm not sure if either of them have it, more likely the latter than the former.

I'll add to this if I find out more. Or @cyrush can add his knowledge when he is back next week.

[yslan]

After digging into the code a bit further, I found vtkh's API vtkh::AboutVTKH(), vtkh::IsCUDAEnabled(), vtkh::ForceCUDA() in src/libs/vtkh/vtkh.hpp which could be useful to double check what backend is actually used.

Otherwise, based on the vtkm code, it will reset the backend if the proposed one is not available.
It is nice to keep the code running. However, I do think user should be aware if the desired backend (says cuda) is not working and the Ascent could actually fall back to OpenMP and introduce performance regression.

Info() from the conduit node prints filters / expressions but not GPU backend.

[yslan]

@nicolemarsaglia and @cyrush

Unfortunately, if I have multiple backends, vtkm will show something like this.
It's still not clear what is used.

---------------- VTK-h -------------------
MPI version: 3.1
VTK-m adapters: Cuda (enabled) Serial (enabled)
------------------------------------------

Let's assume we use vtkh/vtkm.

1. Ascent's vtkh doesn't seem to specify the default backend?

2. What's the default backend for vtkm?
   I got lost when I try to see the source code and their mailing list is not public available.

3. How can we confirm which backend is used without profiling the code?

I appreciate any inputs. Thanks in advance!

[cyrush]

@yslan
If you want to force cuda, you can use:

Node opts;
opts["runtine/vtkm/backend"] = "cuda";
// ... other mpi options, etc
ascent.open(opts);

This will call VTK-h's "Force Cuda" routine, which should configure VTK-m to only allow cuda and not fall back to serial.

ascent/src/libs/ascent/ascent.cpp

Line 273 in 950c5f1

vtkh::ForceCUDA();

If possible, I would still run once with a device profiler to check everything is on device.

[cyrush]

@yslan the best path is to use ascent options - not the vtk-h api directly, so let's make sure whatever you need you can access it via that API.

Argonne folks have been working on testing Intel GPU Support. VTK-m will use kokkos using SYCL, and for the places were we use RAJA that will also be the path.

[yslan]

Then, I need some "feedback" from Ascent's interface like whether the backend is available or whether it's working, such as IsCUDAAvailable and IsCUDAEnable.

[stgeke]

How about ["cuda","Kokkos::HIP","openmp","serial"]? This way you get what you ask for or you fail.

[cyrush]

sounds good, I'll work on the specific names and the expected semantics and create a short write up as an issue

[cyrush]

Here is an issue with some initial ideas to evolve:

#1273