Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Strange metric values when using counter defined in templateDefaults #12821

Open
3 of 4 tasks
reisei opened this issue Mar 19, 2024 · 2 comments · May be fixed by #12833
Open
3 of 4 tasks

Strange metric values when using counter defined in templateDefaults #12821

reisei opened this issue Mar 19, 2024 · 2 comments · May be fixed by #12833
Labels
area/metrics type/bug

Comments

@reisei
Copy link

reisei commented Mar 19, 2024 •
edited by agilgur5

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issue exists when I tested with :latest
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

I want to implement a simple counter of the workflows steps for every workflow. In order to do that I'm using templateDefaults in the workflowDefaults parameter like this:

    # ...
    workflowDefaults:
      metadata:
        labels:
          id: default
      spec:
        templateDefaults:
          metrics:
            prometheus:
            - name: result_counter
              help: "Count of step execution by result status"
              labels:
                - key: name
                  value: "{{workflow.name}}"
                - key: status
                  value: "{{status}}"
              counter:
                value: "1"
        podGC:
          # delete all pods as soon as they complete
          strategy: OnPodCompletion
     # ...

I'm testing this on this example workflow https://github.com/argoproj/argo-workflows/blob/main/examples/workflow-template/dag.yaml

Surprisingly I got wrong counter value in the end of the workflow execution: 2, however in my mind the result should by the amount of pods used in this workflow: 8

argo_workflows_result_counter{name="workflow-template-dag-diamond-z9z4n",status="Succeeded"} 2
Screenshot 2024-03-19 at 11 07 02

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-dag-diamond-
spec:
  entrypoint: diamond
  templates:
  - name: diamond
    dag:
      tasks:
      - name: A
        templateRef:
          name: workflow-template-whalesay-template
          template: whalesay-template
        arguments:
          parameters:
          - name: message
            value: A
      - name: B
        depends: "A"
        templateRef:
          name: workflow-template-whalesay-template
          template: whalesay-template
        arguments:
          parameters:
          - name: message
            value: B
      - name: C
        depends: "A"
        templateRef:
          name: workflow-template-inner-dag
          template: inner-diamond
      - name: D
        depends: "B && C"
        templateRef:
          name: workflow-template-whalesay-template
          template: whalesay-template
        arguments:
          parameters:
          - name: message
            value: D

Logs from the workflow controller

{"Phase":"","ResourceVersion":"38201","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-03-19T09:43:31.885Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":0,"time":"2024-03-19T09:43:31.910Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Updated phase  -\u003e Running","namespace":"workflows","time":"2024-03-19T09:43:31.910Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:43:31.910Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"workflows","time":"2024-03-19T09:43:31.910Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"DAG node workflow-template-dag-diamond-z9z4n initialized Running","namespace":"workflows","time":"2024-03-19T09:43:31.910Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:31.910Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-74501979, taskName B","time":"2024-03-19T09:43:31.910Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-91279598, taskName A","time":"2024-03-19T09:43:31.910Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-91279598, taskName A","time":"2024-03-19T09:43:31.910Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.A dependencies [] completed","namespace":"workflows","time":"2024-03-19T09:43:31.910Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:43:31.910Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Pod node workflow-template-dag-diamond-z9z4n-91279598 initialized Pending","namespace":"workflows","time":"2024-03-19T09:43:31.911Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Created pod: workflow-template-dag-diamond-z9z4n.A (workflow-template-dag-diamond-z9z4n-whalesay-template-91279598)","namespace":"workflows","time":"2024-03-19T09:43:31.924Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-74501979, taskName B","time":"2024-03-19T09:43:31.924Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-57724360, taskName C","time":"2024-03-19T09:43:31.924Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-57724360, taskName C","time":"2024-03-19T09:43:31.924Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:31.924Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-74501979, taskName B","time":"2024-03-19T09:43:31.924Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:31.924Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-03-19T09:43:31.924Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-03-19T09:43:31.924Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"38205","time":"2024-03-19T09:43:31.930Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"Phase":"Running","ResourceVersion":"38205","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-03-19T09:43:41.886Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":1,"time":"2024-03-19T09:43:41.887Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-91279598","time":"2024-03-19T09:43:41.887Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-template-dag-diamond-z9z4n-91279598","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-03-19T09:43:41.887Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:41.887Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-74501979, taskName B","time":"2024-03-19T09:43:41.887Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-74501979, taskName B","time":"2024-03-19T09:43:41.887Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-74501979, taskName B","time":"2024-03-19T09:43:41.887Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.B dependencies [A] completed","namespace":"workflows","time":"2024-03-19T09:43:41.887Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:43:41.887Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Pod node workflow-template-dag-diamond-z9z4n-74501979 initialized Pending","namespace":"workflows","time":"2024-03-19T09:43:41.888Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Created pod: workflow-template-dag-diamond-z9z4n.B (workflow-template-dag-diamond-z9z4n-whalesay-template-74501979)","namespace":"workflows","time":"2024-03-19T09:43:41.895Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-57724360, taskName C","time":"2024-03-19T09:43:41.895Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-57724360, taskName C","time":"2024-03-19T09:43:41.895Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-57724360, taskName C","time":"2024-03-19T09:43:41.895Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.C dependencies [A] completed","namespace":"workflows","time":"2024-03-19T09:43:41.895Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:43:41.895Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"DAG node workflow-template-dag-diamond-z9z4n-57724360 initialized Running","namespace":"workflows","time":"2024-03-19T09:43:41.896Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:43:41.896Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-32178221, taskName inner-B","time":"2024-03-19T09:43:41.896Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4276812660, taskName inner-A","time":"2024-03-19T09:43:41.896Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4276812660, taskName inner-A","time":"2024-03-19T09:43:41.896Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.C.inner-A dependencies [] completed","namespace":"workflows","time":"2024-03-19T09:43:41.896Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:43:41.896Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Pod node workflow-template-dag-diamond-z9z4n-4276812660 initialized Pending","namespace":"workflows","time":"2024-03-19T09:43:41.897Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Created pod: workflow-template-dag-diamond-z9z4n.C.inner-A (workflow-template-dag-diamond-z9z4n-whalesay-template-4276812660)","namespace":"workflows","time":"2024-03-19T09:43:41.904Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-32178221, taskName inner-B","time":"2024-03-19T09:43:41.904Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-15400602, taskName inner-C","time":"2024-03-19T09:43:41.904Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-15400602, taskName inner-C","time":"2024-03-19T09:43:41.904Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:43:41.904Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-32178221, taskName inner-B","time":"2024-03-19T09:43:41.904Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:43:41.904Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:41.904Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:41.904Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-03-19T09:43:41.904Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-03-19T09:43:41.904Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"38247","time":"2024-03-19T09:43:41.916Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-whalesay-template-91279598/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:43:46.917Z"}
{"Phase":"Running","ResourceVersion":"38247","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-03-19T09:43:51.896Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":3,"time":"2024-03-19T09:43:51.897Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-91279598","time":"2024-03-19T09:43:51.897Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-4276812660","time":"2024-03-19T09:43:51.897Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-74501979","time":"2024-03-19T09:43:51.897Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-template-dag-diamond-z9z4n-4276812660","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-03-19T09:43:51.897Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-template-dag-diamond-z9z4n-74501979","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-03-19T09:43:51.897Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:51.897Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:43:51.898Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-32178221, taskName inner-B","time":"2024-03-19T09:43:51.898Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-32178221, taskName inner-B","time":"2024-03-19T09:43:51.898Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-32178221, taskName inner-B","time":"2024-03-19T09:43:51.898Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.C.inner-B dependencies [inner-A] completed","namespace":"workflows","time":"2024-03-19T09:43:51.898Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:43:51.898Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Pod node workflow-template-dag-diamond-z9z4n-32178221 initialized Pending","namespace":"workflows","time":"2024-03-19T09:43:51.898Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Created pod: workflow-template-dag-diamond-z9z4n.C.inner-B (workflow-template-dag-diamond-z9z4n-whalesay-template-32178221)","namespace":"workflows","time":"2024-03-19T09:43:51.902Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-15400602, taskName inner-C","time":"2024-03-19T09:43:51.902Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-15400602, taskName inner-C","time":"2024-03-19T09:43:51.903Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-15400602, taskName inner-C","time":"2024-03-19T09:43:51.903Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.C.inner-C dependencies [inner-A] completed","namespace":"workflows","time":"2024-03-19T09:43:51.903Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:43:51.903Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Pod node workflow-template-dag-diamond-z9z4n-15400602 initialized Pending","namespace":"workflows","time":"2024-03-19T09:43:51.903Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Created pod: workflow-template-dag-diamond-z9z4n.C.inner-C (workflow-template-dag-diamond-z9z4n-whalesay-template-15400602)","namespace":"workflows","time":"2024-03-19T09:43:51.907Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:43:51.907Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:43:51.907Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:51.907Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:43:51.907Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-03-19T09:43:51.907Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-03-19T09:43:51.907Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"38315","time":"2024-03-19T09:43:51.942Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-whalesay-template-74501979/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:43:56.944Z"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-whalesay-template-4276812660/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:43:56.944Z"}
{"Phase":"Running","ResourceVersion":"38315","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-03-19T09:44:01.905Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":5,"time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-32178221","time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-15400602","time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-91279598","time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-4276812660","time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-74501979","time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-template-dag-diamond-z9z4n-32178221","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-template-dag-diamond-z9z4n-15400602","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-03-19T09:44:01.906Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:44:01.906Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:44:01.907Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:44:01.907Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-4226479803, taskName inner-D","time":"2024-03-19T09:44:01.907Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.C.inner-D dependencies [inner-C inner-B] completed","namespace":"workflows","time":"2024-03-19T09:44:01.907Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:44:01.907Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Pod node workflow-template-dag-diamond-z9z4n-4226479803 initialized Pending","namespace":"workflows","time":"2024-03-19T09:44:01.907Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Created pod: workflow-template-dag-diamond-z9z4n.C.inner-D (workflow-template-dag-diamond-z9z4n-whalesay-template-4226479803)","namespace":"workflows","time":"2024-03-19T09:44:01.913Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:44:01.913Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:44:01.913Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-03-19T09:44:01.913Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-03-19T09:44:01.913Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"38378","time":"2024-03-19T09:44:01.919Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-whalesay-template-32178221/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:44:06.921Z"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-whalesay-template-15400602/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:44:06.921Z"}
{"Phase":"Running","ResourceVersion":"38378","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-03-19T09:44:11.914Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":6,"time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-15400602","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-4226479803","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-91279598","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-4276812660","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-74501979","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-32178221","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-template-dag-diamond-z9z4n-4226479803","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:44:11.915Z"}
{"level":"info","msg":"Outbound nodes of workflow-template-dag-diamond-z9z4n-57724360 set to [workflow-template-dag-diamond-z9z4n-4226479803]","namespace":"workflows","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node workflow-template-dag-diamond-z9z4n-57724360 phase Running -\u003e Succeeded","namespace":"workflows","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node workflow-template-dag-diamond-z9z4n-57724360 finished: 2024-03-19 09:44:11.915977437 +0000 UTC","namespace":"workflows","time":"2024-03-19T09:44:11.915Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:44:11.916Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-template-dag-diamond-z9z4n-175167693, taskName D","time":"2024-03-19T09:44:11.916Z"}
{"level":"info","msg":"All of node workflow-template-dag-diamond-z9z4n.D dependencies [B C] completed","namespace":"workflows","time":"2024-03-19T09:44:11.916Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-03-19T09:44:11.916Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Pod node workflow-template-dag-diamond-z9z4n-175167693 initialized Pending","namespace":"workflows","time":"2024-03-19T09:44:11.916Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Created pod: workflow-template-dag-diamond-z9z4n.D (workflow-template-dag-diamond-z9z4n-whalesay-template-175167693)","namespace":"workflows","time":"2024-03-19T09:44:11.922Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-03-19T09:44:11.922Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-03-19T09:44:11.922Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"38424","time":"2024-03-19T09:44:11.929Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-whalesay-template-4226479803/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:44:16.930Z"}
{"Phase":"Running","ResourceVersion":"38424","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-03-19T09:44:21.925Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":7,"time":"2024-03-19T09:44:21.925Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-4226479803","time":"2024-03-19T09:44:21.925Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-175167693","time":"2024-03-19T09:44:21.925Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-91279598","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-4276812660","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-74501979","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-32178221","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-template-dag-diamond-z9z4n-15400602","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-template-dag-diamond-z9z4n-175167693","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Outbound nodes of workflow-template-dag-diamond-z9z4n set to [workflow-template-dag-diamond-z9z4n-175167693]","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node workflow-template-dag-diamond-z9z4n phase Running -\u003e Succeeded","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"node workflow-template-dag-diamond-z9z4n finished: 2024-03-19 09:44:21.926678619 +0000 UTC","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Marking workflow completed","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Marking workflow as pending archiving","namespace":"workflows","time":"2024-03-19T09:44:21.926Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-1340600742-agent/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:44:21.932Z"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Succeeded","resourceVersion":"38466","time":"2024-03-19T09:44:21.934Z","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"archiving workflow","namespace":"workflows","time":"2024-03-19T09:44:21.959Z","uid":"d8fae3d0-0f7b-443e-baa3-36a35f2df339","workflow":"workflow-template-dag-diamond-z9z4n"}
{"level":"info","msg":"Queueing Succeeded workflow workflows/workflow-template-dag-diamond-z9z4n for delete in 1h0m0s due to TTL","time":"2024-03-19T09:44:21.979Z"}
{"action":"deletePod","key":"workflows/workflow-template-dag-diamond-z9z4n-whalesay-template-175167693/deletePod","level":"info","msg":"cleaning up pod","time":"2024-03-19T09:44:26.959Z"}
{"level":"info","msg":"Queueing Succeeded workflow workflows/workflow-template-dag-diamond-z9z4n for delete in 58m28s due to TTL","time":"2024-03-19T09:45:53.533Z"}
{"level":"info","msg":"Queueing Succeeded workflow workflows/workflow-template-dag-diamond-z9z4n for delete in 38m28s due to TTL","time":"2024-03-19T10:05:53.534Z"}

Logs from in your workflow's wait container

Might not get all the logs 

{"Duration":1000000000,"Factor":1.6,"Jitter":0.5,"Steps":5,"level":"info","msg":"Using executor retry strategy","time":"2024-03-19T10:36:20.795Z"}
{"deadline":"0001-01-01T00:00:00Z","includeScriptOutput":false,"level":"info","msg":"Executor initialized","namespace":"workflows","podName":"workflow-template-dag-diamond-z9z4n-whalesay-template-2640029330","templateName":"whalesay-template","time":"2024-03-19T10:36:20.795Z","version":"\u0026Version{Version:v3.5.5,BuildDate:2024-02-29T21:00:43Z,GitCommit:c80b2e91ebd7e7f604e88442f45ec630380effa0,GitTag:v3.5.5,GitTreeState:clean,GoVersion:go1.21.7,Compiler:gc,Platform:linux/arm64,}"}
{"level":"info","msg":"Starting deadline monitor","time":"2024-03-19T10:36:20.803Z"}
{"error":null,"level":"info","msg":"Main container completed","time":"2024-03-19T10:36:22.804Z"}
{"level":"info","msg":"No Script output reference in workflow. Capturing script output ignored","time":"2024-03-19T10:36:22.805Z"}
{"level":"info","msg":"No output parameters","time":"2024-03-19T10:36:22.805Z"}
{"level":"info","msg":"No output artifacts","time":"2024-03-19T10:36:22.805Z"}
{"level":"info","msg":"Alloc=9318 TotalAlloc=13184 Sys=23653 NumGC=3 Goroutines=8","time":"2024-03-19T10:36:22.812Z"}
{"level":"info","msg":"Deadline monitor stopped","time":"2024-03-19T10:36:22.818Z"}
{"error":"context canceled","level":"info","msg":"stopping progress monitor (context done)","time":"2024-03-19T10:36:22.818Z"}
{"level":"info","msg":"Starting Workflow Executor","time":"2024-03-19T10:36:20.809Z","version":"v3.5.5"}
{"Duration":1000000000,"Factor":1.6,"Jitter":0.5,"Steps":5,"level":"info","msg":"Using executor retry strategy","time":"2024-03-19T10:36:20.812Z"}
{"deadline":"0001-01-01T00:00:00Z","includeScriptOutput":false,"level":"info","msg":"Executor initialized","namespace":"workflows","podName":"workflow-template-dag-diamond-z9z4n-whalesay-template-2656806949","templateName":"whalesay-template","time":"2024-03-19T10:36:20.812Z","version":"\u0026Version{Version:v3.5.5,BuildDate:2024-02-29T21:00:43Z,GitCommit:c80b2e91ebd7e7f604e88442f45ec630380effa0,GitTag:v3.5.5,GitTreeState:clean,GoVersion:go1.21.7,Compiler:gc,Platform:linux/arm64,}"}
{"level":"info","msg":"Starting deadline monitor","time":"2024-03-19T10:36:20.817Z"}
{"error":null,"level":"info","msg":"Main container completed","time":"2024-03-19T10:36:22.818Z"}
{"level":"info","msg":"No Script output reference in workflow. Capturing script output ignored","time":"2024-03-19T10:36:22.818Z"}
{"level":"info","msg":"No output parameters","time":"2024-03-19T10:36:22.818Z"}
{"level":"info","msg":"No output artifacts","time":"2024-03-19T10:36:22.818Z"}
{"level":"info","msg":"Alloc=7278 TotalAlloc=13165 Sys=23397 NumGC=4 Goroutines=8","time":"2024-03-19T10:36:22.827Z"}
{"level":"info","msg":"Starting Workflow Executor","time":"2024-03-19T10:36:30.982Z","version":"v3.5.5"}
{"Duration":1000000000,"Factor":1.6,"Jitter":0.5,"Steps":5,"level":"info","msg":"Using executor retry strategy","time":"2024-03-19T10:36:30.984Z"}
{"deadline":"0001-01-01T00:00:00Z","includeScriptOutput":false,"level":"info","msg":"Executor initialized","namespace":"workflows","podName":"workflow-template-dag-diamond-z9z4n-whalesay-template-2556141235","templateName":"whalesay-template","time":"2024-03-19T10:36:30.984Z","version":"\u0026Version{Version:v3.5.5,BuildDate:2024-02-29T21:00:43Z,GitCommit:c80b2e91ebd7e7f604e88442f45ec630380effa0,GitTag:v3.5.5,GitTreeState:clean,GoVersion:go1.21.7,Compiler:gc,Platform:linux/arm64,}"}
{"level":"info","msg":"Starting deadline monitor","time":"2024-03-19T10:36:30.992Z"}
{"error":null,"level":"info","msg":"Main container completed","time":"2024-03-19T10:36:32.994Z"}
{"level":"info","msg":"No Script output reference in workflow. Capturing script output ignored","time":"2024-03-19T10:36:32.994Z"}
{"level":"info","msg":"No output parameters","time":"2024-03-19T10:36:32.994Z"}
{"level":"info","msg":"No output artifacts","time":"2024-03-19T10:36:32.994Z"}
{"level":"info","msg":"Alloc=9190 TotalAlloc=13184 Sys=19557 NumGC=3 Goroutines=8","time":"2024-03-19T10:36:33.000Z"}
{"error":"context canceled","level":"info","msg":"stopping progress monitor (context done)","time":"2024-03-19T10:36:33.003Z"}
@agilgur5 agilgur5 changed the title Strange metric values from the counter defined in the templateDefaults Strange metric values when using counter defined in templateDefaults Mar 19, 2024
@jswxstw
Copy link
Contributor

jswxstw commented Mar 22, 2024 •
edited

Root cause: forgot to merge templateDefaults to the task tmpl.

argo-workflows/workflow/controller/dag.go

Lines 433 to 440 in e547b2d

// Collect the completed task metrics
_, tmpl, _, _ := dagCtx.tmplCtx.ResolveTemplate(task)
if tmpl != nil && tmpl.Metrics != nil {
if prevNodeStatus, ok := woc.preExecutionNodePhases[node.ID]; ok && !prevNodeStatus.Fulfilled() {
localScope, realTimeScope := woc.prepareMetricScope(node)
woc.computeMetrics(tmpl.Metrics.Prometheus, localScope, realTimeScope, false)
}
}

Surprisingly I got wrong counter value in the end of the workflow execution: 2, however in my mind the result should by the amount of pods used in this workflow: 8

By the way, if you use templateDefaults, argo_workflows_result_counter may be 10(DAG templates also count).

jswxstw pushed a commit to jswxstw/argo-workflows that referenced this issue Mar 22, 2024
fix: Merge templateDefaults into dag task tmpl. Fixes argoproj#12821
453d7bf 
Signed-off-by: oninowang <oninowang@tencent.com>
@jswxstw jswxstw linked a pull request Mar 22, 2024 that will close this issue
Open
@reisei
Copy link
Author

reisei commented Mar 25, 2024

Thank you very much for a quick response and the fix!

@jswxstw jswxstw mentioned this issue Apr 9, 2024
Open
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/metrics type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Merge templateDefaults into dag task tmpl. Fixes #12821 jswxstw/argo-workflows
3 participants
@agilgur5 @reisei @jswxstw