Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Daemon pod is always succeeded even with a non-zero status code #12911

Open
3 of 4 tasks
KataaOmar opened this issue Apr 8, 2024 · 4 comments · May be fixed by #12988
Open
3 of 4 tasks

Daemon pod is always succeeded even with a non-zero status code #12911

KataaOmar opened this issue Apr 8, 2024 · 4 comments · May be fixed by #12988
Assignees
@shuangkun
Labels
area/daemon-steps type/bug

Comments

@KataaOmar
Copy link

KataaOmar commented Apr 8, 2024 •
edited

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issue exists when I tested with :latest
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Daemon pod is always succeeded even with a non-zero status code, I expect to find the daemon step failed, then the workflow as well.
image

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: daemon-step-
  namespace: argo
spec:
  entrypoint: daemon-example
  templates:
  - name: daemon-example
    steps:
    - - name: daemon-pod
        template: daemon-temp
                    
    - - name: client-pod             
        template: client-temp 

  - name: daemon-temp
    daemon: true                       
    container:
      image: alpine
      command: ["/bin/sh", "-c"]
      args:
      - |
        sleep 30
        exit 1

  - name: client-temp
    container:
      image: alpine
      command: ["/bin/sh", "-c"]
      args: ["sleep 600"]

Logs from the workflow controller

workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:00:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:13:57.016Z" level=info msg="Alloc=4104 TotalAlloc=167252 Sys=19301 NumGC=447 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:23:00.333Z" level=info msg="Alloc=47122 TotalAlloc=20074390 Sys=142600 NumGC=799 Goroutines=186"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:00.333Z" level=info msg="Alloc=52411 TotalAlloc=20105721 Sys=142600 NumGC=801 Goroutines=186"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:00:22.228Z" level=info msg="Alloc=3956 TotalAlloc=74070 Sys=14949 NumGC=190 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:01:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:02:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:03:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:04:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:05:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:05:22.228Z" level=info msg="Alloc=4131 TotalAlloc=74930 Sys=14949 NumGC=192 Goroutines=18"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:14:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:15:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:16:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:06:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:07:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:08:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:09:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:17:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:18:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:18:57.016Z" level=info msg="Alloc=3965 TotalAlloc=168130 Sys=19301 NumGC=450 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:10:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:10:22.228Z" level=info msg="Alloc=3972 TotalAlloc=75682 Sys=14949 NumGC=195 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:11:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:12:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="Processing workflow" Phase=Failed ResourceVersion=843798350 namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:19:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="Task-result reconciliation" namespace=platform numObjs=8 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-574790004 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.667Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-1281463502 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-1981189896 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:13:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:14:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:15:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:15:22.228Z" level=info msg="Alloc=4208 TotalAlloc=76598 Sys=14949 NumGC=197 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-2640092480 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-3398277239 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-3767785617 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:20:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:21:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:16:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:22:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:17:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:23:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:18:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:23:57.015Z" level=info msg="Alloc=4147 TotalAlloc=169066 Sys=19301 NumGC=452 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:19:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:24:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:25:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:26:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-4209023662 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:20:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.668Z" level=info msg="task-result changed" namespace=platform nodeID=playwright-brt-publish-tcvvj-435495160 workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:20:22.228Z" level=info msg="Alloc=3980 TotalAlloc=77446 Sys=14949 NumGC=200 Goroutines=18"
-fct4n
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:21:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:22:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.669Z" level=info msg="TaskSet Reconciliation" namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:27:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:28:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:23:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:24:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:25:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:28:57.016Z" level=info msg="Alloc=3978 TotalAlloc=169943 Sys=19301 NumGC=455 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.669Z" level=info msg=reconcileAgentPod namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.669Z" level=info msg="Running OnExit handler: exit-handler" namespace=platform workflow=playwright-brt-publish-tcvvj
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.671Z" level=info msg="Workflow to be dehydrated" Workflow Size=51061
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:25:22.229Z" level=info msg="Alloc=4197 TotalAlloc=78389 Sys=14949 NumGC=202 Goroutines=18"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:29:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:30:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:31:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:26:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:27:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:28:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:29:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:30:16.662Z" level=info msg="healthz: current pod is not the leader"

workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:30:22.228Z" level=info msg="Alloc=3946 TotalAlloc=79229 Sys=14949 NumGC=205 Goroutines=18"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:31:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:32:16.663Z" level=info msg="healthz: current pod is not the leader"

workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:28:08.671Z" level=info msg="Workflow to be dehydrated" Workflow Size=36561
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.463Z" level=info msg="Received Workflow Controller config map argo/workflow-controller-configmap update"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.470Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  gcs:\n    bucket: argo-artifact-repository-staging\nexecutor:\n  name: \"\"\n  resources: {}\n  securityContext:\n    allowPrivilegeEscalation: false\n    capabilities:\n      drop:\n      - ALL\n    privileged: false\n    runAsGroup: 8737\n    runAsNonRoot: true\n    runAsUser: 8737\n  terminationMessagePolicy: FallbackToLogsOnError\n  volumeMounts:\n  - mountPath: /tmp\n    name: tmp\ninitialDelay: 0s\nmainContainer:\n  name: \"\"\n  resources: {}\n  securityContext:\n    allowPrivilegeEscalation: false\n    capabilities:\n      drop:\n      - ALL\n    privileged: false\n    readOnlyRootFilesystem: true\n    runAsGroup: 8737\n    runAsNonRoot: true\n    runAsUser: 8737\n  terminationMessagePolicy: FallbackToLogsOnError\n  volumeMounts:\n  - mountPath: /tmp\n    name: tmp\nmetricsConfig: {}\nnodeEvents: {}\npersistence:\n  archive: true\n  archiveTTL: 26280h0m0s\n  clusterName: playground\n  connectionPool:\n    maxIdleConns: 100\n  postgresql:\n    database: postgres\n    host: 10.40.33.7\n    passwordSecret:\n      key: password\n      name: argo-postgres-config\n    port: 5432\n    ssl: true\n    sslMode: require\n    tableName: argo_workflows\n    userNameSecret:\n      key: username\n      name: argo-postgres-config\npodSpecLogStrategy: {}\nsso:\n  clientId:\n    key: client-id\n    name: argo-workflows-sso\n  clientSecret:\n    key: client-secret\n    name: argo-workflows-sso\n  issuer: https://staging.argocd.forbes.com/api/dex\n  rbac:\n    enabled: true\n  redirectUrl: https://staging.argo.forbes.com/oauth2/callback\n  scopes:\n  - groups\n  sessionExpiry: 0s\ntelemetryConfig: {}\nworkflowDefaults:\n  metadata:\n    creationTimestamp: null\n  spec:\n    activeDeadlineSeconds: 18000\n    arguments: {}\n    automountServiceAccountToken: false\n    executor:\n      serviceAccountName: argo-read\n    podDisruptionBudget:\n      maxUnavailable: 0\n    podGC:\n      strategy: OnWorkflowSuccess\n    securityContext:\n      runAsGroup: 8737\n      runAsNonRoot: true\n      runAsUser: 8737\n    serviceAccountName: argo-read\n    templateDefaults:\n      inputs: {}\n      metadata: {}\n      outputs: {}\n      volumes:\n      - emptyDir: {}\n        name: tmp\n    ttlStrategy:\n      secondsAfterFailure: 10800\n      secondsAfterSuccess: 900\n  status:\n    finishedAt: null\n    startedAt: null\n"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.470Z" level=info msg="Persistence configuration enabled"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.471Z" level=info msg="Node status offloading is disabled"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.471Z" level=info msg="Workflow archiving is enabled"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:29:14.471Z" level=info executorImage="quay.io/argoproj/argoexec:v3.5.5" executorImagePullPolicy= managedNamespace=
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.001Z" level=info msg="Running clear-non-running-pods" namespace=argo workflow=clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.001Z" level=info msg="Running cert-checker-error" namespace=argo workflow=cert-checker-error
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.001Z" level=info msg="clear-non-running-pods is suspended, skipping execution" namespace=argo workflow=clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:30:00.002Z" level=info msg="cert-checker-error is suspended, skipping execution" namespace=argo workflow=cert-checker-error
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:32:33.116Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:00.334Z" level=info msg="Alloc=47471 TotalAlloc=20171583 Sys=142600 NumGC=804 Goroutines=187"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:33:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:33:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.948Z" level=info msg="Processing workflow" Phase= ResourceVersion=902005365 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.955Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="Created PDB resource for workflow." namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="Steps node daemon-step-qw2b6 initialized Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.968Z" level=info msg="StepGroup node daemon-step-qw2b6-84771817 initialized Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.969Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:54.969Z" level=info msg="Pod node daemon-step-qw2b6-707072755 initialized Pending" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="Created pod: daemon-step-qw2b6[0].daemon-pod (daemon-step-qw2b6-daemon-temp-707072755)" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="Workflow step group node daemon-step-qw2b6-84771817 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.021Z" level=info msg="Workflow to be dehydrated" Workflow Size=1706
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:33:55.032Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005371 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:33:57.016Z" level=info msg="Alloc=4175 TotalAlloc=170827 Sys=19301 NumGC=457 Goroutines=18"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.022Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005371 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.023Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="Node became daemoned" namespace=argo nodeId=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=daemon-step-qw2b6-707072755 old.message= old.phase=Pending old.progress=0/1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.024Z" level=info msg="Step group node daemon-step-qw2b6-84771817 successful" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="node daemon-step-qw2b6-84771817 phase Running -> Succeeded" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="node daemon-step-qw2b6-84771817 finished: 2024-04-08 09:34:05.02501423 +0000 UTC" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="StepGroup node daemon-step-qw2b6-1091576052 initialized Running" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.025Z" level=info msg="Pod node daemon-step-qw2b6-3790013067 initialized Pending" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="Created pod: daemon-step-qw2b6[1].client-pod (daemon-step-qw2b6-client-temp-3790013067)" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.064Z" level=info msg="Workflow to be dehydrated" Workflow Size=2620
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:05.079Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005503 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.064Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005503 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.065Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=daemon-step-qw2b6-3790013067 old.message= old.phase=Pending old.progress=0/1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.066Z" level=info msg="Workflow to be dehydrated" Workflow Size=2692
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:15.078Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005629 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:34:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.079Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005629 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.080Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.081Z" level=info msg="Workflow to be dehydrated" Workflow Size=2693
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:25.095Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005749 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:34:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.095Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005749 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=daemon-step-qw2b6-707072755 old.message= old.phase=Running old.progress=0/1 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.097Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.098Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:35.111Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005874 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.112Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005874 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.113Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.114Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:34:45.124Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005874 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:35:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:35:22.228Z" level=info msg="Alloc=4146 TotalAlloc=80063 Sys=14949 NumGC=207 Goroutines=18"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:35:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.662Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005874 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.663Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.664Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.676Z" level=info msg="Workflow to be dehydrated" Workflow Size=51061
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:36:04.676Z" level=info msg="Workflow to be dehydrated" Workflow Size=36561
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:36:16.663Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:36:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:37:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:37:33.117Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:00.333Z" level=info msg="Alloc=74797 TotalAlloc=20233272 Sys=142600 NumGC=806 Goroutines=188"
workflow-controller-6dfc6d9c44-mdzfq time="2024-04-08T09:38:16.662Z" level=info msg="healthz: current pod is not the leader"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:20.231Z" level=info msg="Executor plugin updated" name=slack-plugin namespace=argo
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing qa/bq-validator" cronWorkflow=qa/bq-validator
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing backend/update-forbesapi" cronWorkflow=backend/update-forbesapi
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing content-products/bertie-bulletin-2k" cronWorkflow=content-products/bertie-bulletin-2k
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing frontend/update-simple-site-staging-deployments" cronWorkflow=frontend/update-simple-site-staging-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.063Z" level=info msg="CronWorkflow qa/bq-validator added" cronWorkflow=qa/bq-validator
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing argo/clear-non-running-pods" cronWorkflow=argo/clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.064Z" level=info msg="CronWorkflow argo/clear-non-running-pods added" cronWorkflow=argo/clear-non-running-pods
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing content-products/bertie-bulletin-po" cronWorkflow=content-products/bertie-bulletin-po
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.065Z" level=info msg="Processing argo/cert-checker-error" cronWorkflow=argo/cert-checker-error
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.068Z" level=info msg="CronWorkflow argo/cert-checker-error added" cronWorkflow=argo/cert-checker-error
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.069Z" level=info msg="Processing argo/fastly-google-oauth" cronWorkflow=argo/fastly-google-oauth
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.071Z" level=info msg="CronWorkflow argo/fastly-google-oauth added" cronWorkflow=argo/fastly-google-oauth
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing profile-portal-dev/staging-profile-api-mongoexport" cronWorkflow=profile-portal-dev/staging-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.073Z" level=info msg="Processing argo/gcp-dns-backup" cronWorkflow=argo/gcp-dns-backup
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.075Z" level=info msg="CronWorkflow argo/gcp-dns-backup added" cronWorkflow=argo/gcp-dns-backup
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.075Z" level=info msg="Processing argo/update-simple-site-buckets" cronWorkflow=argo/update-simple-site-buckets
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.085Z" level=info msg="Processing content-products/bertie-bulletin" cronWorkflow=content-products/bertie-bulletin
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.052Z" level=info msg="Processing frontend/update-simple-site-production-deployments" cronWorkflow=frontend/update-simple-site-production-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.132Z" level=info msg="CronWorkflow content-products/bertie-bulletin-po added" cronWorkflow=content-products/bertie-bulletin-po
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.138Z" level=info msg="Processing argo/cert-sync" cronWorkflow=argo/cert-sync
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.139Z" level=info msg="CronWorkflow content-products/bertie-bulletin added" cronWorkflow=content-products/bertie-bulletin
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.139Z" level=info msg="Processing argo/find-unhealthy-deployments" cronWorkflow=argo/find-unhealthy-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.132Z" level=info msg="CronWorkflow profile-portal-dev/staging-profile-api-mongoexport added" cronWorkflow=profile-portal-dev/staging-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.140Z" level=info msg="Processing argo/cert-checker-warn" cronWorkflow=argo/cert-checker-warn
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.143Z" level=info msg="CronWorkflow argo/find-unhealthy-deployments added" cronWorkflow=argo/find-unhealthy-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.144Z" level=info msg="Processing mobile/daily-mobile-check" cronWorkflow=mobile/daily-mobile-check
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.153Z" level=info msg="CronWorkflow argo/cert-checker-warn added" cronWorkflow=argo/cert-checker-warn
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.161Z" level=info msg="Processing profile-portal-dev/production-profile-api-mongoexport" cronWorkflow=profile-portal-dev/production-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.165Z" level=info msg="CronWorkflow argo/update-simple-site-buckets added" cronWorkflow=argo/update-simple-site-buckets
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.169Z" level=info msg="CronWorkflow mobile/daily-mobile-check added" cronWorkflow=mobile/daily-mobile-check
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.161Z" level=info msg="CronWorkflow argo/cert-sync added" cronWorkflow=argo/cert-sync
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.161Z" level=info msg="CronWorkflow content-products/bertie-bulletin-2k added" cronWorkflow=content-products/bertie-bulletin-2k
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.177Z" level=info msg="CronWorkflow profile-portal-dev/production-profile-api-mongoexport added" cronWorkflow=profile-portal-dev/production-profile-api-mongoexport
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.194Z" level=error msg="cannot validate Workflow: templates.update-forbesapi-staging-deployments.steps[0].update-deloyment templates.update-argocd-appset-deployment-main.steps[2].cicd templates.cicd inputs.parameters.team was not supplied" conditionType=SpecError namespace=backend workflow=update-forbesapi
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.194Z" level=error msg="invalid cron workflow" cronWorkflow=backend/update-forbesapi error="cannot validate Workflow: templates.update-forbesapi-staging-deployments.steps[0].update-deloyment templates.update-argocd-appset-deployment-main.steps[2].cicd templates.cicd inputs.parameters.team was not supplied"
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.267Z" level=info msg="CronWorkflow frontend/update-simple-site-staging-deployments added" cronWorkflow=frontend/update-simple-site-staging-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:21.286Z" level=info msg="CronWorkflow frontend/update-simple-site-production-deployments added" cronWorkflow=frontend/update-simple-site-production-deployments
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.274Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=902005874 namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.275Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="task-result changed" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-3790013067 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="node unchanged" namespace=argo nodeID=daemon-step-qw2b6-707072755 workflow=daemon-step-qw2b6
-fct4n
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.276Z" level=info msg="SG Outbound nodes of daemon-step-qw2b6-707072755 are [daemon-step-qw2b6-707072755]" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg="Workflow step group node daemon-step-qw2b6-1091576052 not yet completed" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg=reconcileAgentPod namespace=argo workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.277Z" level=info msg="Workflow to be dehydrated" Workflow Size=2803

workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.280Z" level=info msg="Workflow to be dehydrated" Workflow Size=36561h-tcvvj-1340600742-agent/deletePod
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.291Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=902005874 workflow=daemon-step-qw2b6
workflow-controller-6dfc6d9c44-7zrrc time="2024-04-08T09:38:30.309Z" level=info msg="Workflow update successful" namespace=pref-center-dev phase=Running resourceVersion=836386165 workflow=pref-center-
workflow-controller-6dfc6d9c44-9rhcw time="2024-04-08T09:38:33.117Z" level=info msg="healthz: current pod is not the leader"

Logs from in your workflow's wait container

time="2024-04-08T13:17:31.231Z" level=info msg="Starting Workflow Executor" version=v3.5.5
time="2024-04-08T13:17:31.235Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-04-08T13:17:31.235Z" level=info msg="Executor initialized" deadline="2024-04-08 18:17:29 +0000 UTC" includeScriptOutput=false namespace=argo podName=daemon-step-566zz-daemon-temp-1963735462 templateName=daemon-temp version="&Version{Version:v3.5.5,BuildDate:2024-02-29T20:59:20Z,GitCommit:c80b2e91ebd7e7f604e88442f45ec630380effa0,GitTag:v3.5.5,GitTreeState:clean,GoVersion:go1.21.7,Compiler:gc,Platform:linux/amd64,}"
W0408 13:17:31.249886       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
time="2024-04-08T13:17:31.250Z" level=info msg="Starting deadline monitor"
time="2024-04-08T13:18:02.286Z" level=info msg="Main container completed" error="<nil>"
time="2024-04-08T13:18:02.286Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-04-08T13:18:02.286Z" level=info msg="No output parameters"
time="2024-04-08T13:18:02.286Z" level=info msg="No output artifacts"
time="2024-04-08T13:18:02.287Z" level=info msg="GCS Save path: /tmp/argo/outputs/logs/main.log, key: daemon-step-566zz/daemon-step-566zz-daemon-temp-1963735462/main.log"
time="2024-04-08T13:18:02.538Z" level=info msg="Save artifact" artifactName=main-logs duration=250.989995ms error="<nil>" key=daemon-step-566zz/daemon-step-566zz-daemon-temp-1963735462/main.log
time="2024-04-08T13:18:02.538Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-04-08T13:18:02.538Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
W0408 13:18:02.551995       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
W0408 13:18:02.559278       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
time="2024-04-08T13:18:02.559Z" level=info msg="Alloc=26115 TotalAlloc=32548 Sys=39781 NumGC=5 Goroutines=11"
W0408 13:18:02.566281       1 warnings.go:70] Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
Stream closed EOF for argo/daemon-step-566zz-daemon-temp-1963735462 (wait)
@shuangkun shuangkun self-assigned this Apr 8, 2024
@shuangkun
Copy link
Member

The current design indeed ignores daemon failed.

argo-workflows/workflow/controller/operator.go

Line 1336 in 934bf17

// ignore pod failure for daemoned steps

@KataaOmar
Copy link
Author

@shuangkun
Is there a specific purpose for that? And what should we do in this case?

@artem-zherdiev-ingio
Copy link

artem-zherdiev-ingio commented Apr 25, 2024 •
edited

We also have a similar issue, how we can catch daemon failure via hook to send notifications? We have the same in DAG templates.

shuangkun added a commit to shuangkun/argo-workflows that referenced this issue Apr 27, 2024
@shuangkun
fix: set daemon pod failed when with a non-zero status. Fixes:argopro…
0d68499 
…j#12911

Signed-off-by: shuangkun <tsk2013uestc@163.com>
@shuangkun shuangkun linked a pull request Apr 27, 2024 that will close this issue
Open
shuangkun added a commit to shuangkun/argo-workflows that referenced this issue Apr 27, 2024
@shuangkun
fix: set daemon pod failed when with a non-zero status. Fixes:argopro…
3a9d764 
…j#12911

Signed-off-by: shuangkun <tsk2013uestc@163.com>
shuangkun added a commit to shuangkun/argo-workflows that referenced this issue Apr 27, 2024
@shuangkun
fix: set daemon pod failed when with a non-zero status. Fixes:argopro…
1724e0b 
…j#12911

Signed-off-by: shuangkun <tsk2013uestc@163.com>
shuangkun added a commit to shuangkun/argo-workflows that referenced this issue May 8, 2024
@shuangkun
fix: set daemon pod failed when with a non-zero status. Fixes:argopro…
9a2fcfd 
…j#12911

Signed-off-by: shuangkun <tsk2013uestc@163.com>
shuangkun added a commit to shuangkun/argo-workflows that referenced this issue May 8, 2024
@shuangkun
fix: set daemon pod failed when with a non-zero status. Fixes:argopro…
5c83ec3 
…j#12911

Signed-off-by: shuangkun <tsk2013uestc@163.com>
shuangkun added a commit to shuangkun/argo-workflows that referenced this issue May 9, 2024
@shuangkun
fix: set daemon pod failed when with a non-zero status. Fixes:argopro…
0eebc2d 
…j#12911

Signed-off-by: shuangkun <tsk2013uestc@163.com>
@juliev0
Copy link
Contributor

juliev0 commented May 10, 2024

FYI, just added this comment in the PR: #12988 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shuangkun shuangkun

Labels
area/daemon-steps type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: set daemon pod failed when exit with a non-zero status. Fixes:#12911 shuangkun/argo-workflows
4 participants
@juliev0 @shuangkun @KataaOmar @artem-zherdiev-ingio