Inline Workflows logs in UI always empty

[artem-zherdiev-ingio]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issue exists when I tested with :latest
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

Inline Workflows always show an empty log in UI, it doesn't matter if the task succeeded or not. We are still able to read logs from the pod in the console.

Expectation: inline workflows should show correct logs in UI.

Version

v3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: inline-cant-show-errors
  namespace: workflows
spec:
  serviceAccountName: workflows-executor

  podGC:
    strategy: OnWorkflowSuccess

  entrypoint: pipeline
  templates:
    - name: pipeline
      dag:
        tasks:
          - name: broken-configmap
            inline:
              serviceAccountName: configmap-owner
              resource:
                action: get
                manifest: |
                  apiVersion: v1
                  kind: ConfigMap
                  metadata:
                    name: non-existing-configmap
                    namespace: workflows
              outputs:
                parameters:
                  - name: value
                    valueFrom:
                      default: "default-value"
                      jqFilter: '.data["value"]'
          - name: correct-shell
            inline:
              container:
                image: alpine:3.7
                command: [sh, -c]
                args: ["echo hello"]
          - name: broken-shell
            inline:
              container:
                image: alpine:3.7
                command: [sh, -c]
                args: ["cat /tmp/non-existing-file"]

Logs from the workflow controller

time="2024-04-05T12:04:45.437Z" level=info msg="Processing workflow" Phase= ResourceVersion=285687282 namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.459Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=0 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.459Z" level=info msg="Updated phase  -> Running" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.463Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.463Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.463Z" level=info msg="DAG node inline-cant-show-errors-ppxjk initialized Running" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.463Z" level=warning msg="was unable to obtain the node for inline-cant-show-errors-ppxjk-1239447401, taskName broken-configmap"
time="2024-04-05T12:04:45.463Z" level=warning msg="was unable to obtain the node for inline-cant-show-errors-ppxjk-1239447401, taskName broken-configmap"
time="2024-04-05T12:04:45.463Z" level=info msg="All of node inline-cant-show-errors-ppxjk.broken-configmap dependencies [] completed" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.463Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.463Z" level=info msg="Pod node inline-cant-show-errors-ppxjk-1239447401 initialized Pending" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.510Z" level=info msg="Created pod: inline-cant-show-errors-ppxjk.broken-configmap (inline-cant-show-errors-ppxjk--1239447401)" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.510Z" level=warning msg="was unable to obtain the node for inline-cant-show-errors-ppxjk-897640899, taskName broken-shell"
time="2024-04-05T12:04:45.510Z" level=warning msg="was unable to obtain the node for inline-cant-show-errors-ppxjk-897640899, taskName broken-shell"
time="2024-04-05T12:04:45.510Z" level=info msg="All of node inline-cant-show-errors-ppxjk.broken-shell dependencies [] completed" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.510Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.510Z" level=info msg="Pod node inline-cant-show-errors-ppxjk-897640899 initialized Pending" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.559Z" level=info msg="Created pod: inline-cant-show-errors-ppxjk.broken-shell (inline-cant-show-errors-ppxjk--897640899)" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.559Z" level=warning msg="was unable to obtain the node for inline-cant-show-errors-ppxjk-109496782, taskName correct-shell"
time="2024-04-05T12:04:45.559Z" level=warning msg="was unable to obtain the node for inline-cant-show-errors-ppxjk-109496782, taskName correct-shell"
time="2024-04-05T12:04:45.560Z" level=info msg="All of node inline-cant-show-errors-ppxjk.correct-shell dependencies [] completed" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.560Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.560Z" level=info msg="Pod node inline-cant-show-errors-ppxjk-109496782 initialized Pending" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.612Z" level=info msg="Created pod: inline-cant-show-errors-ppxjk.correct-shell (inline-cant-show-errors-ppxjk--109496782)" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.612Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.612Z" level=info msg=reconcileAgentPod namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:45.625Z" level=info msg="Workflow update successful" namespace=workflows phase=Running resourceVersion=285687291 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.510Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=285687291 namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=3 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="task-result changed" namespace=workflows nodeID=inline-cant-show-errors-ppxjk-897640899 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="task-result changed" namespace=workflows nodeID=inline-cant-show-errors-ppxjk-1239447401 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="task-result changed" namespace=workflows nodeID=inline-cant-show-errors-ppxjk-109496782 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="node changed" namespace=workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=inline-cant-show-errors-ppxjk-109496782 old.message= old.phase=Pending old.progress=0/1 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="Pod failed: Error (exit code 1)" displayName=broken-configmap namespace=workflows pod=inline-cant-show-errors-ppxjk--1239447401 templateName= workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=inline-cant-show-errors-ppxjk-1239447401 old.message= old.phase=Pending old.progress=0/1 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="Pod failed: Error (exit code 1)" displayName=broken-shell namespace=workflows pod=inline-cant-show-errors-ppxjk--897640899 templateName= workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.511Z" level=info msg="node changed" namespace=workflows new.message="Error (exit code 1)" new.phase=Failed new.progress=0/1 nodeID=inline-cant-show-errors-ppxjk-897640899 old.message= old.phase=Pending old.progress=0/1 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.512Z" level=info msg="Outbound nodes of inline-cant-show-errors-ppxjk set to [inline-cant-show-errors-ppxjk-1239447401 inline-cant-show-errors-ppxjk-897640899 inline-cant-show-errors-ppxjk-109496782]" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.512Z" level=info msg="node inline-cant-show-errors-ppxjk phase Running -> Failed" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.512Z" level=info msg="node inline-cant-show-errors-ppxjk finished: 2024-04-05 12:04:55.512049976 +0000 UTC" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.512Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.512Z" level=info msg=reconcileAgentPod namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.512Z" level=info msg="Updated phase Running -> Failed" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.518Z" level=info msg="cleaning up pod" action=deletePod key=workflows/inline-cant-show-errors-ppxjk-1340600742-agent/deletePod
time="2024-04-05T12:04:55.525Z" level=info msg="Workflow update successful" namespace=workflows phase=Failed resourceVersion=285687597 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:04:55.532Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/inline-cant-show-errors-ppxjk--1239447401/labelPodCompleted
time="2024-04-05T12:04:55.532Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/inline-cant-show-errors-ppxjk--897640899/labelPodCompleted
time="2024-04-05T12:04:55.532Z" level=info msg="cleaning up pod" action=labelPodCompleted key=workflows/inline-cant-show-errors-ppxjk--109496782/labelPodCompleted
time="2024-04-05T12:05:05.581Z" level=info msg="Processing workflow" Phase=Failed ResourceVersion=285687597 namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:05:05.583Z" level=info msg="Task-result reconciliation" namespace=workflows numObjs=3 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:05:05.583Z" level=info msg="task-result changed" namespace=workflows nodeID=inline-cant-show-errors-ppxjk-897640899 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:05:05.583Z" level=info msg="task-result changed" namespace=workflows nodeID=inline-cant-show-errors-ppxjk-1239447401 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:05:05.583Z" level=info msg="task-result changed" namespace=workflows nodeID=inline-cant-show-errors-ppxjk-109496782 workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:05:05.583Z" level=info msg="TaskSet Reconciliation" namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:05:05.583Z" level=info msg=reconcileAgentPod namespace=workflows workflow=inline-cant-show-errors-ppxjk
time="2024-04-05T12:05:05.589Z" level=info msg="cleaning up pod" action=deletePod key=workflows/inline-cant-show-errors-ppxjk-1340600742-agent/deletePod
time="2024-04-05T12:05:05.597Z" level=info msg="Workflow update successful" namespace=workflows phase=Failed resourceVersion=285687597 workflow=inline-cant-show-errors-ppxjk

Logs from in your workflow's wait container

-