Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: setBucketLifecycleRule error in OSS Artifact Driver. Fixes #12925 #12926

Merged
merged 2 commits into from May 4, 2024

Conversation

AlbeeSo
Copy link
Contributor

@AlbeeSo AlbeeSo commented Apr 11, 2024

Fixes #12925

Motivation

OSS does not support to set different rules at a time for a same prefix and that's why I received a 400 error. And more details added in related issue.

time="2024-04-09T16:04:23.147Z" level=info msg="Save artifact" artifactName=out duration=1.270302632s error="oss: service returned error: StatusCode=400, ErrorCode=InvalidRequest, ErrorMessage=\"Found same prefix in different rules\", RequestId=6615670711BF5732361C11E2, Ec=0014-00000085" key=argo-workflows/test/lifecycletime="2024-04-09T16:04:23.147Z" level=error msg="executor error: oss: service returned error: StatusCode=400, ErrorCode=InvalidRequest, ErrorMessage=\"Found same prefix in different rules\", RequestId=6615670711BF5732361C11E2, Ec=0014-00000085"

Modifications

LifecycleRule modified to:

  • Set Transitions with MarkInfrequentAccessAfterDays (Objects will be converted to IA)
  • Set Expiration with MarkDeletionAfterDays. (Objects will be deleted if Versioning is disabled, or changed to non-current version or else)
  • Set AbortMultipartUpload with MarkDeletionAfterDays also. (Aborted Uploaded Parts will be deleted)

Verification

Please test with workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: set-lifecycle-
spec:
  entrypoint: main
  templates:
    - name: main
      metadata:
        annotations:
          k8s.aliyun.com/eci-extra-ephemeral-storage: "5Gi"
          k8s.aliyun.com/eci-use-specs : "ecs.g7.xlarge"
      container:
        image: alpine:latest
        command:
          - sh
          - -c
        args:
          - |
            mkdir -p /out
            dd if=/dev/random of=/out/lifecycle.txt bs=2M count=1024
            echo "created files!"
      outputs:
        artifacts:
          - name: out
            path: /out/lifecycle.txt
            oss:
              endpoint: http://oss-cn-zhangjiakou-internal.aliyuncs.com
              bucket: my-argo-workflow
              key: argo-workflows/test/lifecycle
              accessKeySecret:
                name: my-argo-workflow-credentials
                key: accessKey
              secretKeySecret:
                name: my-argo-workflow-credentials
                key: secretKey
              lifecycleRule:
                markInfrequentAccessAfterDays: 1
                markDeletionAfterDays: 3

@AlbeeSo
Copy link
Contributor Author

AlbeeSo commented Apr 11, 2024

image
And here's my test result.

@AlbeeSo AlbeeSo marked this pull request as draft April 11, 2024 17:22
…SS Artifact Driver.

Signed-off-by: AlbeeSo <suyashi1321@163.com>
@AlbeeSo AlbeeSo marked this pull request as ready for review April 11, 2024 17:24
NonVersionTransitions: []oss.LifecycleVersionTransition{versionTransition},

rule := oss.LifecycleRule{
ID: "argo-workflows-rule",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this ID need to correspond to artifacts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OSS ArtifactDriver could only receive the name of key and setting of OSSBucket, so I used sha256(key) as ID which could avoid exceeding length limit.

Signed-off-by: AlbeeSo <suyashi1321@163.com>
@shuangkun shuangkun self-assigned this Apr 11, 2024
Copy link
Member

@shuangkun shuangkun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shuangkun shuangkun added the area/artifacts S3/GCP/OSS/Git/HDFS etc label Apr 15, 2024
@agilgur5 agilgur5 changed the title fix: modified lifecycle rule to fix setBucketLifecycleRule error in OSS Artifact Drive. Fixes #12925 fix: setBucketLifecycleRule error in OSS Artifact Driver. Fixes #12925 May 4, 2024
@terrytangyuan terrytangyuan merged commit a4fc318 into argoproj:main May 4, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc
Projects
None yet
Development

Successfully merging this pull request may close these issues.

oss artifact: get 400 error from OSS server when OSSLifecycleRule is set
3 participants