Error in Kubeflow Pipelines: open /var/run/argo/outputs/parameters/tmp/outputs/execution-id: no such file or directory

[nehiridil]

I am building a pipeline in Kubeflow pipelines on k3s in my WSL2 Ubuntu Machine
kfp 2.0.1
kfp-kubernetes 1.0.0
kfp-pipeline-spec 0.2.2
kfp-server-api 2.0.5

argo: v3.5.2
  BuildDate: 2023-11-27T19:06:22Z
  GitCommit: 5b6ad2be163ecd3f0251a931ab84dba3c6085ad2
  GitTreeState: clean
  GitTag: v3.5.2
  GoVersion: go1.21.4
  Compiler: gc
  Platform: linux/amd64

I am getting this error in a simple hello world pipeline: /var/run/argo/outputs/parameters/tmp/outputs/execution-id
Do you have any clue about the reason of this error? Any help would be appreciated

Containers:
  wait:
    Container ID:  containerd://01af60845cebc9a18383c90a0e4978dd0da98c7e141d8cd6980b31566162640f
    Image:         quay.io/argoproj/argoexec:v3.4.11
    Image ID:      quay.io/argoproj/argoexec@sha256:6731738dc79454232937c73128ccf6a9e74ffbdd0a613fb427ae427ee482115b
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
      --loglevel
      info
      --log-format
      text
    State:          Terminated
      Reason:       Error
      Message:      open /var/run/argo/outputs/parameters/tmp/outputs/execution-id: no such file or directory
      Exit Code:    1
      Started:      Mon, 08 Jan 2024 13:23:06 +0100
      Finished:     Mon, 08 Jan 2024 13:23:08 +0100
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:                      hello-pipeline-fs8jj-system-dag-driver-3921009115 (v1:metadata.name)
      ARGO_POD_UID:                        (v1:metadata.uid)
      GODEBUG:                            x509ignoreCN=0
      ARGO_WORKFLOW_NAME:                 hello-pipeline-fs8jj
      ARGO_CONTAINER_NAME:                wait
      ARGO_TEMPLATE:                      {"name":"system-dag-driver","inputs":{"parameters":[{"name":"component","value":"{\"dag\":{\"outputs\":{\"parameters\":{\"Output\":{\"valueFromParameter\":{\"outputParameterKey\":\"Output\",\"producerSubtask\":\"say-hello\"}}}},\"tasks\":{\"say-hello\":{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-say-hello\"},\"inputs\":{\"parameters\":{\"name\":{\"componentInputParameter\":\"recipient\"}}},\"taskInfo\":{\"name\":\"say-hello\"}}}},\"inputDefinitions\":{\"parameters\":{\"recipient\":{\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"parameters\":{\"Output\":{\"parameterType\":\"STRING\"}}}}"},{"name":"runtime-config","default":"","value":"{\"parameterValues\":{\"recipient\":\"Jane\"}}"},{"name":"task","default":"","value":""},{"name":"parent-dag-id","default":"0","value":"0"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"driver-type","default":"DAG","value":"ROOT_DAG"}]},"outputs":{"parameters":[{"name":"execution-id","valueFrom":{"path":"/tmp/outputs/execution-id"}},{"name":"iteration-count","valueFrom":{"path":"/tmp/outputs/iteration-count","default":"0"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:8e60086b04d92b657898a310ca9757631d58547e76bbbb8bfc376d654bef1707","command":["driver"],"args":["--type","ROOT_DAG","--pipeline_name","hello-pipeline","--run_id","de140679-1cd5-4555-85b0-460f553dc6ef","--dag_execution_id","0","--component","{\"dag\":{\"outputs\":{\"parameters\":{\"Output\":{\"valueFromParameter\":{\"outputParameterKey\":\"Output\",\"producerSubtask\":\"say-hello\"}}}},\"tasks\":{\"say-hello\":{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-say-hello\"},\"inputs\":{\"parameters\":{\"name\":{\"componentInputParameter\":\"recipient\"}}},\"taskInfo\":{\"name\":\"say-hello\"}}}},\"inputDefinitions\":{\"parameters\":{\"recipient\":{\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"parameters\":{\"Output\":{\"parameterType\":\"STRING\"}}}}","--task","","--runtime_config","{\"parameterValues\":{\"recipient\":\"Jane\"}}","--iteration_index","-1","--execution_id_path","/tmp/outputs/execution-id","--iteration_count_path","/tmp/outputs/iteration-count","--condition_path","/tmp/outputs/condition"],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}}}
      ARGO_NODE_ID:                       hello-pipeline-fs8jj-3921009115
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      0001-01-01T00:00:00Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
    Mounts:
      /tmp from tmp-dir-argo (rw,path="0")
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-npnv7 (ro)

[agilgur5]

Image: quay.io/argoproj/argoexec:v3.4.11

argo: v3.5.2

Your container says you're running 3.4.11, not 3.5.2 as you wrote.

In either case though, Kubeflow Pipelines is still on an outdated Argo version, 3.3 (see also kubeflow/pipelines#9301 et al), so this could be a result of some of those incompatibilities (I've never seen this error before).

It also looks like you posted this same question in kubeflow/pipelines#10365 and resolved it recently?

[nehiridil]

Yes it resolved a couple of hours ago, I upgraded argo version to 3.5.3 today and changed the deployment file from install.yaml to quick-start-minimal.yaml because I am using k3s.