feat: implement OpenStream func of OSS artifactdriver. Part of #8489

[AlbeeSo]

Part of #8489

Motivation

OSS Artifact Driver do not support OpenStream Func to avoid unnecessary file downloading.

Modifications

Implemented OpenStream Func of OSS Artifact Driver.

Verification

Get an artifact which storages in OSS from UI, check the logs of argo-server.
latest version:

time="2024-04-07T13:03:58.700Z" level=info msg="Efficient artifact streaming is not supported for type *oss.ArtifactDriver: see https://github.com/argoproj/argo-workflows/issues/8489"
time="2024-04-07T13:04:10.072Z" level=info msg="Efficient artifact streaming is not supported for type *oss.ArtifactDriver: see https://github.com/argoproj/argo-workflows/issues/8489"

submitted version:

time="2024-04-07T13:22:29.074Z" level=info msg="OSS OpenStream, key: argo-workflows/input-artifact-oss-k4htb/input-artifact-oss-k4htb/main.log"
time="2024-04-07T13:22:29.092Z" level=info msg="Stream artifact" artifactName=main-logs duration=18.139213ms error="<nil>" key=argo-workflows/input-artifact-oss-k4htb/input-artifact-oss-k4htb/main.log

[shuangkun]

Could you add some info in doc：
https://argo-workflows.readthedocs.io/en/latest/configure-artifact-repository/#artifact-streaming

[shuangkun]

Could you add some info in doc： https://argo-workflows.readthedocs.io/en/latest/configure-artifact-repository/#artifact-streaming

Maybe can add it after it is released.

[terrytangyuan]

Can we implement this in the same PR?

[AlbeeSo]

It seems that OSS SDK (or S3 also) does not currently support operations like "GetObjectsWithPrefix" to a .tgz file. So the only way might be to list objects with prefix first, load, tar them then and stream the .tgz file to user finally, more like a "LoadToStream" way.
https://github.com/argoproj/argo-workflows/blob/34967127bfe304da0f72dc1dba63558a1609b2d0/server/artifacts/artifact_server.go#L184C1-L233C36
Maybe it would be better to still return an html page?

[terrytangyuan]

Can you test if a relatively large artifact can be viewed successfully as well?

[AlbeeSo]

Can you test if a relatively large artifact can be viewed successfully as well?

I tested with workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-
spec:
  entrypoint: main
  templates:
    - name: main
      container:
        image: alpine:latest
        command:
          - sh
          - -c
        args:
          - |
            mkdir -p /out
            dd if=/dev/random of=/out/testfile.txt bs=20M count=1024
            echo "Download!"
      outputs:
        artifacts:
          - name: out
            path: /out/testfile.txt
            oss:
              endpoint: http://oss-cn-zhangjiakou-internal.aliyuncs.com
              bucket: bucket
              key: argo-workflows/test/bigfile
              accessKeySecret:
                name: my-argo-workflow-credentials
                key: accessKey
              secretKeySecret:
                name: my-argo-workflow-credentials
                key: secretKey

to download a 20GB output file from UI successfully.
Then, I used a video around 3GB instead which could be played normally after downloading. During this period, the memory usage of argo server is stable.

[AlbeeSo]

Could you add some info in doc： https://argo-workflows.readthedocs.io/en/latest/configure-artifact-repository/#artifact-streaming

Maybe can add it after it is released.

Sorry Im little bit confused about the support version, is it v3.5+?

[shuangkun]

Could you add some info in doc： https://argo-workflows.readthedocs.io/en/latest/configure-artifact-repository/#artifact-streaming

Maybe can add it after it is released.

Sorry Im little bit confused about the support version, is it v3.5+?

No need to add it now. Please add it when a new version is released.

[shuangkun]

you can change title reference #11823

[agilgur5]

No need to add it now. Please add it when a new version is released.

We should try to get docs in ASAP with features, ideally within the same PR. The longer we wait the more likely that something is either undocumented or documented incorrectly.

Sorry Im little bit confused about the support version, is it v3.5+?

It would be v3.6+.

We have versioned documentation now (#11390) and already have various docs for new features coming to v3.6 with appropriate version specifiers (#12581)