Reuse existing task pods

[porbmv83]

Instead of creating a new pod per task Argo could look if there is an existing pod available and use it. I guess this would increase performance because it wouldn't need to create it again.

[Charlie17Li]

Are there any plans to support this feature? 🤔

[evgeniy-khatko]

Is there any progress on this issue?
Looks like important scaling feature. Is there a workaround or supported solution?

I can imagine in most cases a workflow would pull a docker image and execute it. The pulling and image building is an overhead that can be avoided by reusing workflow pods? Am I missing smth obvious?

[sarabala1979]

You can use containerSet template. It will use the same pod to execute the all steps.

[marcstreeter]

the problem here is with potential wasted costs as all containers are running even when not in use

[pisymbol]

A really great idea would be if one could specify a pool of pods (a workflow pod pool) and these pods would be kept "hot" for workflows to use.

For example, let's say I have a task that requires a large ML model to be loaded into memory (which can take minutes depending on the size), and I have several workflows and/or tasks within a workflow that all use it. What I want is that Pod to be kept "hot" (running) while workflows are ran instead of each workflow spawning the same Pod over and over again and incurring the initial startup/scheduling costs over and over again.

The container template would work for some use cases (keep inference steps together maybe) but comes with a lot of limitations.

[jswxstw]

The solution we adopted is to use HTTP/Plugin Template instead of Container Template.
This allows all steps to be executed in the same agent pod.

[dineshkumar20]

@jswxstw Could you please explain how it will use same agent pod?

[jswxstw]

You can refer to the following workflow to compare the differences and time consumption between running with shell and http templates.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: http-benchmark-
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: step1
            template: http
            arguments:
              parameters: [{name: url, value: "https://raw.githubusercontent.com/argoproj/argo-workflows/4e450e250168e6b4d51a126b784e90b11a0162bc/pkg/apis/workflow/v1alpha1/generated.swagger.json"}]
        - - name: step2
            template: http
            arguments:
              parameters: [{name: url, value: "https://raw.githubusercontent.com/argoproj/argo-workflows/4e450e250168e6b4d51a126b784e90b11a0162bc/pkg/apis/workflow/v1alpha1/generated.swagger.json"}]
        - - name: step3
            template: http
            arguments:
              parameters: [{name: url, value: "https://raw.githubusercontent.com/argoproj/argo-workflows/4e450e250168e6b4d51a126b784e90b11a0162bc/pkg/apis/workflow/v1alpha1/generated.swagger.json"}]
    - name: shell
      inputs:
        parameters:
          - name: url
      container:
        image: docker/whalesay:latest
        command: [ sh, -c ]
        args: [ "echo '{{inputs.parameters.url}}'" ]
    - name: http
      inputs:
        parameters:
          - name: url
      http:
        url: "{{inputs.parameters.url}}"

[dineshkumar20]

Thank you for sharing the details. This approach will be helpful when we make HTTP requests from each step. In my case, all steps will process some files and generate reports. Would it be possible to reuse the same pod?

[jswxstw]

If you are interested in how this workflow works, you can refer to the documentation: argo-agent.
In simple terms, each workflow will only start one agent pod at most. For HTTP/Plugin templates, the controller will generate tasks and write them into WorkflowTaskSet. The worker goroutines in the agent pod are responsible for executing these tasks, and the node results will be written back to WorkflowTaskSet.

[jswxstw]

For complex steps, you can consider using Plugins.

[shuangkun]

Do you want to use a service that is already running in the cluster, such as Vincent diagram, and then all workflows will call this service, and then have some input and get some output?

Probably you can use this: https://argo-workflows.readthedocs.io/en/latest/async-pattern/
Define your input and output rules, and define two pods in argo, one to trigger and one to wait. Avoid wasting resources creating Pods for this service.
We have many scenarios where we call an emr job or interact with the slurm cluster in the argo workflow.