[FEATURE]: "deploy" kubernetes-pod Run and Test action types

[salotz]

Feature Request

Background / Motivation

I commonly find myself using a pattern of first creating a Deployment, via garden kubernetes deploy action, and then running things on it with kubernetes-exec (run and test action types).

Shouldn't I just use kubernetes-pod then?

There are basically 2 answers to this:

1. Often times kubernetes-pod deployments do not function like the kubernetes deployment (volumes not mounting, and so on), (fixed in #6112 )
2. I frequently want to get interactive access to my deployments for either debugging or just ad hoc testing and development.

This post is just about 2.

A concrete (minimally simplified) example:

---
kind: Deploy
name: acceptance-tests-deploy
type: kubernetes
description: |
  Deployment of acceptance test ready pod. In addition to test
  environment being the same as the normal tests deployment it also
  mounts the test data volume.

dependencies:
  - deploy.db-config
  - build.tests-container
  - deploy.data-pvc

spec:
  files:
    - test-runner_deploy.yaml

  defaultTarget:
    kind: Deployment
    name: acceptance-tests-deployment

---
kind: Test
name: functional-tests
type: kubernetes-exec
description: |
  Action to run functional tests.

dependencies:
  - deploy.acceptance-tests-deploy

spec:
  command:
    - nox
    - -s
    - test_functional
  resource:
    kind: Deployment
    name: acceptance-tests-deployment

---
kind: Test
name: acceptance-tests
type: kubernetes-exec
description: |
  Action to run acceptance tests.

dependencies:
  - deploy.acceptance-tests-deploy
  - deploy.api-server
  - run.db-init
  - test.preflight-db-init-tests

include: []

spec:
  command:
    - nox
    - -s
    - test_acceptance
  resource:
    kind: Deployment
    name: acceptance-tests-deployment

I get an actual persistent deployment I can garden exec to for running commands or debugging in.

Now the problems:

1. Typically in a CI environment I don't want this deployment hanging around. I can always add a cleanup task for the deployments (and often do) but CI is hard I would get rid of it if I could.
2. Lots of extra Garden actions to write and maintain.
3. In this example because I have two test actions (that should run in parallel) using the same deployment this can lead to problems in the pod if they are writing/reading from the same data. Which wouldn't be a problem if they had their own pod. If I was to solve this I would end up with another deploy action for each which is also not desirable.

What should the user be able to do?

So I started thinking that wouldn't it be nice if I could just do:

---
kind: Test
name: acceptance-tests
type: kubernetes-pod
[...]

And then only when I want to get a persistent deployment of this thing I could run it with a special flag that would deploy it and keep it until I clean it up. For instance, locally I'm debugging these tests and I run:

garden test --deploy acceptance-tests

The Deployment would get deployed to the cluster and use a patched in command (e.g. sleep infinity)

Why do they want to do this? What problem does it solve?

The main pain point is debugging Run and Test action types running in K8s pods.

Secondarily it is avoiding spinning up long-lived resources on the cluster unnecessarily.

Suggested Implementation(s)

Non-feature: Using ConfigTemplate

I could work around this using a ConfigTemplate which combines a Deploy and a Test/Run action into a single definition. I would have to understand the templated naming conventions of each though.

Specialize Run & Test definitions

Add a flag garden run --deploy myaction that will try to deploy the action. By default it replaces the command with sleep infinity.

Then add extra options for configuring the "deployment" that you would find under a Deploy/kubernetes config. E.g. port forwarding, sync options, command override, etc.

Derive Run/Test kubernetes-pod actions from a Deploy/kubernetes action

In this option you would still write the Deploy and Run/Test action separately as in my demonstration above, but there would be an option to reference the deployment action directly rather than a file or manifest for the action. E.g.

---
Kind: Deploy
name: test-deploy
type: kubernetes
[...]
---

Kind: Test
name: test-a
type: kubernetes-pod

spec:
  fromDeploy:
    action: test-deploy
    file: test_deploy.yaml
  resource:
    type: Deployment
    name: test-deploy-pod

Then when you run:

garden test test-a

It will figure out how to create the pod from the Deploy action, deploy, run the test, and then shut it down.

Then you have the option to either:

garden deploy test-deploy

or this, which might be able to override some things from the referenced deployment:

garden test --deploy test-a

To get something you can exec into or otherwise debug.

How important is this feature for you/your team?

🌹 It’s a nice to have, but nice things are nice 🙂

[salotz]

Updated as it seems part of the problem I had withkubernetes-pod actions has been fixed in: #6112

[thsig]

Hi @salotz!

Thanks for writing this up. We've had similar thoughts a few times in the past, that the kubernetes-exec and kubernetes-pod actions might be easier to use as separate modes of the same action type.

For now, I don't think this will be prioritised, but I'd love to get something like this in sooner or later.