S3 artifact input files have altered timestamps with new version of Argo

[Christoffer-Cortes]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issue exists when I tested with :latest
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

A team in my organization is using Autotools to build their binaries.
Previously they built their binary in one step and used the support in Argo workflows for s3 artifacts to store them.
Then they would use the s3 artifact input in other steps to reuse the already compiled binary.
This worked fine in Argo v3.3.10 but after an upgrade to Argo v3.4.10 and also with Argo v3.5.5, which was the latest helm chart version at the time I was testing, this stopped working right.
The make command has a mechanism to check that the binary files are newer than source code files based on the last modified timestamp, then determines if it needs to recompile the source code or not.

After some investigation I found that this issue: #7486, removed some binaries for internal use in Argo and among those was the tar binary.
The standard library "common/gzip" is now used but this library seems to produce a different modify timestamp compared to how it was previously.

I did some simple artifact output and input test to see if I could see some discrepancies in the timestamps.
I found that the input artifact now has a modified timestamp that seems to match when the file was downloaded and untarred, instead of one that more closely aligns with when it was first created as an output.
The expectation here is that the this modify timestamp would be similar to how it was before to avoid running into problems with the make command.

Logs from Argo v3.3.10

Step: generate-artifact
Container: main

Mon Mar 25 12:16:06 UTC 2024

hello world

  File: hello-world.txt
  Size: 12              Blocks: 8          IO Block: 4096   regular file
Device: 10007eh/1048702d        Inode: 3164435     Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-03-25 12:16:06.000000000
Modify: 2024-03-25 12:16:06.000000000
Change: 2024-03-25 12:16:06.000000000

Container: wait

time="2024-03-25T12:16:08.484Z" level=info msg="Saving output artifacts"
time="2024-03-25T12:16:08.484Z" level=info msg="Staging artifact: hello-world"
time="2024-03-25T12:16:08.484Z" level=info msg="Copying hello-world.txt from container base image layer to /tmp/argo/outputs/artifacts/hello-world.tgz"
time="2024-03-25T12:16:08.484Z" level=info msg="/var/run/argo/outputs/artifacts/hello-world.txt.tgz -> /tmp/argo/outputs/artifacts/hello-world.tgz"
time="2024-03-25T12:16:08.484Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/hello-world.tgz, key: artifacts/2024/03/25/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac-951696058/hello-world.tgz"
time="2024-03-25T12:16:08.484Z" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.seli.gic.ericsson.se:10444"
time="2024-03-25T12:16:08.484Z" level=info msg="Saving file to s3" bucket=docklindev-moore031 endpoint="seliis3obj.seli.gic.ericsson.se:10444" key=artifacts/2024/03/25/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac-951696058/hello-world.tgz path=/tmp/argo/outputs/artifacts/hello-world.tgz
time="2024-03-25T12:16:08.625Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/hello-world.tgz
time="2024-03-25T12:16:08.625Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/hello-world.tgz"

Step: consume-artifact
Container: main

Mon Mar 25 12:16:16 UTC 2024

hello world

  File: hello-world.txt
  Size: 12              Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 3813484     Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-03-25 12:16:16.000000000
Modify: 2024-03-25 12:16:07.000000000
Change: 2024-03-25 12:16:15.000000000

Container: init

time="2024-03-25T12:16:15.901Z" level=info msg="Start loading input artifacts..."
time="2024-03-25T12:16:15.901Z" level=info msg="Downloading artifact: hello-world"
time="2024-03-25T12:16:15.901Z" level=info msg="S3 Load path: /argo/inputs/artifacts/hello-world.tmp, key: artifacts/2024/03/25/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac-951696058/hello-world.tgz"
time="2024-03-25T12:16:15.901Z" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.seli.gic.ericsson.se:10444"
time="2024-03-25T12:16:15.901Z" level=info msg="Getting file from s3" bucket=docklindev-moore031 endpoint="seliis3obj.seli.gic.ericsson.se:10444" key=artifacts/2024/03/25/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac/62b9b3c0-a8f3-47cd-82e6-7b539f3d00ac-951696058/hello-world.tgz path=/argo/inputs/artifacts/hello-world.tmp
time="2024-03-25T12:16:15.965Z" level=info msg="Detecting if /argo/inputs/artifacts/hello-world.tmp is a tarball"
time="2024-03-25T12:16:15.966Z" level=info msg="tar -xf /argo/inputs/artifacts/hello-world.tmp -C /argo/inputs/artifacts/hello-world.tmpdir"
time="2024-03-25T12:16:15.968Z" level=info msg="Successfully download file: /argo/inputs/artifacts/hello-world"

Logs from Argo v3.5.5

Step: generate-artifact
Container: main

Thu Mar 28 09:00:37 UTC 2024

hello world

  File: hello-world.txt
  Size: 12              Blocks: 8          IO Block: 4096   regular file
Device: 10007eh/1048702d        Inode: 2637340     Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-03-28 09:00:37.000000000
Modify: 2024-03-28 09:00:37.000000000
Change: 2024-03-28 09:00:37.000000000

Container: wait

time="2024-03-28T09:00:39 UTC" level=info msg="Saving output artifacts"
time="2024-03-28T09:00:39 UTC" level=info msg="Staging artifact: hello-world"
time="2024-03-28T09:00:39 UTC" level=info msg="Copying hello-world.txt from container base image layer to /tmp/argo/outputs/artifacts/hello-world.tgz"
time="2024-03-28T09:00:39 UTC" level=info msg="/var/run/argo/outputs/artifacts/hello-world.txt.tgz -> /tmp/argo/outputs/artifacts/hello-world.tgz"
time="2024-03-28T09:00:39 UTC" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/hello-world.tgz, key: artifacts/2024/03/28/d5909538-b11f-4166-9cbc-e5b089a5e713/d5909538-b11f-4166-9cbc-e5b089a5e713-say-hello-1294651765/hello-world.tgz"
time="2024-03-28T09:00:39 UTC" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.seli.gic.ericsson.se:10444"
time="2024-03-28T09:00:39 UTC" level=info msg="Saving file to s3" bucket=docklindev-moore031 endpoint="seliis3obj.seli.gic.ericsson.se:10444" key=artifacts/2024/03/28/d5909538-b11f-4166-9cbc-e5b089a5e713/d5909538-b11f-4166-9cbc-e5b089a5e713-say-hello-1294651765/hello-world.tgz path=/tmp/argo/outputs/artifacts/hello-world.tgz
time="2024-03-28T09:00:39 UTC" level=info msg="Save artifact" artifactName=hello-world duration=89.751036ms error="<nil>" key=artifacts/2024/03/28/d5909538-b11f-4166-9cbc-e5b089a5e713/d5909538-b11f-4166-9cbc-e5b089a5e713-say-hello-1294651765/hello-world.tgz
time="2024-03-28T09:00:39 UTC" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/hello-world.tgz
time="2024-03-28T09:00:39 UTC" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/hello-world.tgz"

Step: consume-artifact
Container: main

Thu Mar 28 09:00:47 UTC 2024

hello world

  File: hello-world.txt
  Size: 12              Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 2378954     Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-03-28 09:00:47.000000000
Modify: 2024-03-28 09:00:45.000000000
Change: 2024-03-28 09:00:45.000000000

Container: init

time="2024-03-28T09:00:45 UTC" level=info msg="Start loading input artifacts..."
time="2024-03-28T09:00:45 UTC" level=info msg="Downloading artifact: hello-world"
time="2024-03-28T09:00:45 UTC" level=info msg="S3 Load path: /argo/inputs/artifacts/hello-world.tmp, key: artifacts/2024/03/28/d5909538-b11f-4166-9cbc-e5b089a5e713/d5909538-b11f-4166-9cbc-e5b089a5e713-say-hello-1294651765/hello-world.tgz"
time="2024-03-28T09:00:45 UTC" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.seli.gic.ericsson.se:10444"
time="2024-03-28T09:00:45 UTC" level=info msg="Getting file from s3" bucket=docklindev-moore031 endpoint="seliis3obj.seli.gic.ericsson.se:10444" key=artifacts/2024/03/28/d5909538-b11f-4166-9cbc-e5b089a5e713/d5909538-b11f-4166-9cbc-e5b089a5e713-say-hello-1294651765/hello-world.tgz path=/argo/inputs/artifacts/hello-world.tmp
time="2024-03-28T09:00:45 UTC" level=info msg="Load artifact" artifactName=hello-world duration=48.434132ms error="<nil>" key=artifacts/2024/03/28/d5909538-b11f-4166-9cbc-e5b089a5e713/d5909538-b11f-4166-9cbc-e5b089a5e713-say-hello-1294651765/hello-world.tgz
time="2024-03-28T09:00:45 UTC" level=info msg="Detecting if /argo/inputs/artifacts/hello-world.tmp is a tarball"
time="2024-03-28T09:00:45 UTC" level=info msg="Successfully download file: /argo/inputs/artifacts/hello-world"

Version

3.5.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

kind: Workflow
metadata:
  generateName: s3-artifact-
spec:
  entrypoint: s3-artifact-example
  templates:
  - name: s3-artifact-example
    steps:
    - - name: generate-artifact
        template: say-hello
    - - name: consume-artifact
        template: print-hello-world
        arguments:
          artifacts:
          - name: hello-world
            from: "{{steps.generate-artifact.outputs.artifacts.hello-world}}"
  - name: say-hello
    container:
      image: alpine:latest
      command: [sh, -c]
      args:
      - |
        date
        printf "\n"
        echo "hello world" | tee hello-world.txt
        printf "\n"
        stat hello-world.txt
    outputs:
      artifacts:
      - name: hello-world
        path: hello-world.txt
  - name: print-hello-world
    inputs:
      artifacts:
      - name: hello-world
        path: hello-world.txt
    container:
      image: alpine:latest
      command: [sh, -c]
      args:
      - |
        date
        printf "\n"
        cat hello-world.txt
        printf "\n"
        stat hello-world.txt

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded