Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WorkflowTemplate podMetadata not working #12945

Open
3 of 4 tasks
gnadaban opened this issue Apr 16, 2024 · 2 comments · May be fixed by #12987
Open
3 of 4 tasks

WorkflowTemplate podMetadata not working #12945

gnadaban opened this issue Apr 16, 2024 · 2 comments · May be fixed by #12987
Assignees
@shuangkun
Labels
area/workflow-templates type/bug

Comments

@gnadaban
Copy link

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issue exists when I tested with :latest
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

When podMetadata is specified in a ClusterWorkflowTemplate and the template is invoked from a Workflow instance, the pods are not getting the annotations and labels.

I'm not familiar with the code but the #6512 change seems to have replaced one behavior with another.
Since both Workflow and (Cluster)WorkflowTemplate resources have the spec.podMetadata field, users rightfully expect that they should be able to specify ClusterWorkflowTemplate-level annotations and labels for pods instead of repeating the same information in eg. every container template within that ClusterWorkflowTemplate.

In addition, when the ClusterWorkflowTemplate is invoked from a Workflow, these annotations and labels should be possible to override, so IMO the Workflow spec.podMetadata should take precedence.

I'm not sure if this is correct, but perhaps in workflow/controller/workflowpod.go something like this would work:

// addMetadata applies metadata specified in the template
func (woc *wfOperationCtx) addMetadata(pod *apiv1.Pod, tmpl *wfv1.Template) {
	// Support WorkflowTemplate-level podMetadata 
	workflowMetadata := woc.wf.Spec.PodMetadata
	if executingWorkflowMetadata != nil {
		for k, v := range workflowMetadata.Annotations {
			pod.ObjectMeta.Annotations[k] = v
		}
		for k, v := range workflowMetadata.Labels {
			pod.ObjectMeta.Labels[k] = v
		}
	}

	// Workflow-level podMetadata takes precedence over WorkflowTemplate-level podMetadata
	executingWorkflowMetadata := woc.execWf.Spec.PodMetadata
	if executingWorkflowMetadata != nil {
		// add workflow-level pod annotations and labels
		for k, v := range executingWorkflowMetadata.Annotations {
			pod.ObjectMeta.Annotations[k] = v
		}
		for k, v := range executingWorkflowMetadata.Labels {
			pod.ObjectMeta.Labels[k] = v
		}
	}

	for k, v := range tmpl.Metadata.Annotations {
		pod.ObjectMeta.Annotations[k] = v
	}
	for k, v := range tmpl.Metadata.Labels {
		pod.ObjectMeta.Labels[k] = v
	}
}

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: workflow-template
spec:
  entrypoint: whalesay-template

  podMetadata:
    labels:
      workflow-template-label: hello
    annotations:
      all-pods-should-have-this: value

  arguments:
    parameters:
      - name: message
        value: hello world

  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: test-workflow
  namespace: argo-workflows-system
spec:
  podMetadata:
    labels:
      caller-label: hello

  entrypoint: start
  templates:
    - name: start
      steps:
        - - name: hello
            templateRef:
              name: workflow-template
              template: whalesay-template
              clusterScope: true
            arguments:
              parameters:
                - name: message
                  value: Hello Bug

Logs from the workflow controller

{"Phase":"","ResourceVersion":"74873485","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:36.947Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":0,"time":"2024-04-16T13:40:36.970Z","workflow":"test-workflow"}
{"level":"info","msg":"Updated phase  -\u003e Running","namespace":"argo","time":"2024-04-16T13:40:36.970Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:36.970Z","workflow":"test-workflow"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"Steps node test-workflow initialized Running","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"StepGroup node test-workflow-3484831391 initialized Running","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"Pod node test-workflow-2831459201 initialized Pending","namespace":"argo","time":"2024-04-16T13:40:36.971Z","workflow":"test-workflow"}
{"level":"info","msg":"Created pod: test-workflow[0].hello (test-workflow-whalesay-template-2831459201)","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow step group node test-workflow-3484831391 not yet completed","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:37.075Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow update successful","namespace":"argo","phase":"Running","resourceVersion":"74873494","time":"2024-04-16T13:40:37.092Z","workflow":"test-workflow"}
{"Phase":"","ResourceVersion":"74873605","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:41.910Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":0,"time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Updated phase  -\u003e Running","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Steps node test-workflow initialized Running","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"StepGroup node test-workflow-3484831391 initialized Running","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Pod node test-workflow-2831459201 initialized Pending","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow step group node test-workflow-3484831391 not yet completed","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:41.940Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow update successful","namespace":"argo","phase":"Running","resourceVersion":"74873609","time":"2024-04-16T13:40:41.957Z","workflow":"test-workflow"}
{"Phase":"Running","ResourceVersion":"74873609","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:47.072Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":0,"time":"2024-04-16T13:40:47.073Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow pod is missing","namespace":"argo","nodeName":"test-workflow[0].hello","nodePhase":"Pending","recentlyStarted":true,"time":"2024-04-16T13:40:47.073Z","workflow":"test-workflow"}
{"level":"info","msg":"Created pod: test-workflow[0].hello (test-workflow-whalesay-template-2831459201)","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"level":"info","msg":"Workflow step group node test-workflow-3484831391 not yet completed","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:47.171Z","workflow":"test-workflow"}
{"Phase":"Running","ResourceVersion":"74873609","level":"info","msg":"Processing workflow","namespace":"argo","time":"2024-04-16T13:40:57.172Z","workflow":"test-workflow"}
{"level":"info","msg":"Task-result reconciliation","namespace":"argo","numObjs":1,"time":"2024-04-16T13:40:57.173Z","workflow":"test-workflow"}
{"level":"info","msg":"task-result changed","namespace":"argo","nodeID":"test-workflow-2831459201","time":"2024-04-16T13:40:57.173Z","workflow":"test-workflow"}
{"level":"info","msg":"node changed","namespace":"argo","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"test-workflow-2831459201","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-04-16T13:40:57.173Z","workflow":"test-workflow"}
{"level":"info","msg":"Step group node test-workflow-3484831391 successful","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow-3484831391 phase Running -\u003e Succeeded","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow-3484831391 finished: 2024-04-16 13:40:57.174058552 +0000 UTC","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Outbound nodes of test-workflow-2831459201 is [test-workflow-2831459201]","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Outbound nodes of test-workflow is [test-workflow-2831459201]","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow phase Running -\u003e Succeeded","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"node test-workflow finished: 2024-04-16 13:40:57.174184294 +0000 UTC","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"reconcileAgentPod","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"level":"info","msg":"Marking workflow completed","namespace":"argo","time":"2024-04-16T13:40:57.174Z","workflow":"test-workflow"}
{"action":"deletePod","key":"argo/test-workflow-1340600742-agent/deletePod","level":"info","msg":"cleaning up pod","time":"2024-04-16T13:40:57.179Z"}
{"level":"info","msg":"Workflow update successful","namespace":"argo","phase":"Succeeded","resourceVersion":"74873926","time":"2024-04-16T13:40:57.267Z","workflow":"test-workflow"}
{"action":"labelPodCompleted","key":"argo/test-workflow-whalesay-template-2831459201/labelPodCompleted","level":"info","msg":"cleaning up pod","time":"2024-04-16T13:40:57.291Z"}

Logs from in your workflow's wait container

{"level":"info","msg":"Starting Workflow Executor","time":"2024-04-16T13:40:48.883Z","version":"v3.5.5"}
{"Duration":1000000000,"Factor":1.6,"Jitter":0.5,"Steps":5,"level":"info","msg":"Using executor retry strategy","time":"2024-04-16T13:40:48.886Z"}
{"deadline":"0001-01-01T00:00:00Z","includeScriptOutput":false,"level":"info","msg":"Executor initialized","namespace":"argo","podName":"test-workflow-whalesay-template-2831459201","templateName":"whalesay-template","time":"2024-04-16T13:40:48.886Z","version":"\u0026Version{Version:v3.5.5,BuildDate:2024-02-29T20:59:20Z,GitCommit:c80b2e91ebd7e7f604e88442f45ec630380effa0,GitTag:v3.5.5,GitTreeState:clean,GoVersion:go1.21.7,Compiler:gc,Platform:linux/amd64,}"}
{"level":"info","msg":"Starting deadline monitor","time":"2024-04-16T13:40:48.905Z"}
{"error":null,"level":"info","msg":"Main container completed","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"No Script output reference in workflow. Capturing script output ignored","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"No output parameters","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"No output artifacts","time":"2024-04-16T13:40:50.906Z"}
{"level":"info","msg":"Alloc=7172 TotalAlloc=13211 Sys=23653 NumGC=4 Goroutines=8","time":"2024-04-16T13:40:50.932Z"}
{"level":"info","msg":"Deadline monitor stopped","time":"2024-04-16T13:40:50.944Z"}
@jswxstw
Copy link
Contributor

jswxstw commented Apr 19, 2024

This issue is very similar to #12729, just the opposite.
The difference lies in whether the ClusterWorkflowTemplate is used as a WFT or a WF.
If you use the ClusterWorkflowTemplate as a WFT, you can add the metadata into template whalesay-template.
Howerver, I think it may also be necessary to merge the spec of woc.wf and woc.execWf.

@shuangkun
Copy link
Member

shuangkun commented Apr 20, 2024 •
edited

Does the first executingWorkflowMetadata!=nil should be workflowMetadata? @gnadaban

@shuangkun shuangkun self-assigned this Apr 20, 2024
shuangkun added a commit to shuangkun/argo-workflows that referenced this issue Apr 27, 2024
@shuangkun
fix: set template metadata from workflow template PodMetadata. Fixes:a…
1cf9ac4 
…rgoproj#12945

Signed-off-by: shuangkun <tsk2013uestc@163.com>
@shuangkun shuangkun linked a pull request Apr 27, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shuangkun shuangkun

Labels
area/workflow-templates type/bug
Projects
None yet
Milestone
No milestone
3 participants
@jswxstw @gnadaban @shuangkun