how to mount volume to the 'wait' container?

[tooptoop4]

i'm trying to setup https://argoproj.github.io/argo-workflows/configure-archive-logs/ to write logs to s3
I'm using ISRA on k8s

but the 'wait' container is showing this error:

time="2022-01-04T09:09:03.074Z" level=info msg="Processing workflow" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Pod failed: Error (exit code 1): failed to create new S3 client: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" displayName=aws-sqs-workflow-chr22 namespace=auth pod=aws-sqs-workflow-chr22 templateName=whalesay workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Updating node aws-sqs-workflow-chr22 exit code 0" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Updating node aws-sqs-workflow-chr22 status Running -> Error" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Updating node aws-sqs-workflow-chr22 message: Error (exit code 1): failed to create new S3 client: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="TaskSet Reconciliation" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg=reconcileAgentPod namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Updated phase Running -> Error" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Updated message  -> Error (exit code 1): failed to create new S3 client: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Marking workflow completed" namespace=auth workflow=aws-sqs-workflow-chr22
time="2022-01-04T09:09:03.075Z" level=info msg="Marking workflow as pending archiving" namespace=auth workflow=aws-sqs-workflow-chr22

i used suggestion in #7461 (reply in thread) and can verify the volumes/env are appearing on my main container (looking at kubectl describe po) but not the wait container.

part of my configmap:

artifactRepository:
  archiveLogs: true
  s3:
    bucket: redact
    endpoint: s3.amazonaws.com
    insecure: false
    keyFormat: "argo_logs/"
    region: eu-west-1
    roleARN: arn:aws:iam::redact:role/redactrole
    useSDKCreds: true

looking at https://github.com/argoproj/argo-helm/blob/master/charts/argo-workflows/values.yaml#L253 there is section for env but not volumes

[terrytangyuan]

You can patch your containers via podSpecPatch in your workflow, e.g. https://github.com/argoproj/argo-workflows/blob/master/examples/pod-spec-patch.yaml#L13

Another solution is to customize your executor container spec in the controller configmap: https://github.com/argoproj/argo-workflows/blob/master/docs/workflow-controller-configmap.yaml#L190-L213

[tooptoop4]

@terrytangyuan are there any examples? #4623 is saying its not supported

[terrytangyuan]

Yeah that solution is not for volumes then. Could you try the second approach I mentioned? Basically you can treat executor as a container spec and customize it from there.

[tooptoop4]

@terrytangyuan there is partial success. Using

    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /mainctrfs/var/run/secrets/eks.amazonaws.com/serviceaccount/token

i was able to get the main logs written to s3 but the main logs are not visible in the UI after pod completion (they show while pod is running but after it shows failed to create new S3 client: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors). just the 'wait' logs are visible in the UI.

[terrytangyuan]

Please file a separate issue as it's unrelated to the original question.

[tooptoop4]

@terrytangyuan #7513

[Woytek-Polnik]

None of the suggestions would work.
The updating executor container could allow to update volumeMounts but still does not allow to edit the volumes itself for the workflow pod