integration testing

[edublancas]

Our testing suite is currently limited. For example, when using the Argo integration, we only check that the generated argo YAML spec is valid, but we don't check that it'll run. Furthermore, we skip building the docker image, which is a critical part of the export process: if we make mistakes in the Dockerfile template, chances are we won't notice. This has severely impacted our iteration speed since we have to manually test before each release.

In #109, I proposed writing our tutorials as Jupyter notebooks. This has an added benefit that the examples will be easily runnable, and we can incorporate them to the CI.

However, testing this tutorials is challenging since they depend on infrastructures such as a Kubernetes cluster, SLURM, or AWS Batch. Some observations:

• Looks like we can install kind (hence, install a kubernetes cluster) on github actions
• AWS Batch is the most important one to test; however, our existing tests are good enough, thanks to moto, we can simulate all the infrastructure, although it'd be ideal to test it with real infrastructure.
• the SLURM example already contains a template that spins up a local SLURM cluster, if we can run docker in Github Actions, we can test this (although SLURM isn't heavily used to it's not high priority)

As a first step, we can start testing kubernetes-based examples. AWS BAtch is already reasonably well-tested, and SLURM isn't a priority.