Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check if we could replace blobxfer by azcopy #3414

Closed
lemeurherve opened this issue Feb 27, 2023 · 49 comments
Closed

Check if we could replace blobxfer by azcopy #3414

lemeurherve opened this issue Feb 27, 2023 · 49 comments
Assignees
@lemeurherve
Labels
blobxfer get.jenkins.io javadoc jenkins.io site:archives.jenkins.io Related to this infrastructure service trusted.ci.jenkins.io update-center
Milestone
infra-team-sync-2...

Comments

@lemeurherve
Copy link
Member

lemeurherve commented Feb 27, 2023 •
edited

As we've encountered some issues with blobxfer recently (#3411), and as its last release is quite dated, check if we could replace it with an az-cli command like what's done in the pipeline library: https://github.com/jenkins-infra/pipeline-library/blob/93b13be5d876d90d8cd145b11c9f9fe457239db9/vars/publishReports.groovy#L55-L59

Related:
@dduportal dduportal mentioned this issue Feb 27, 2023
Closed
@lemeurherve lemeurherve mentioned this issue Feb 28, 2023
Closed
@dduportal dduportal removed this from the infra-team-sync-2023-03-07 milestone Mar 8, 2023
@lemeurherve lemeurherve mentioned this issue Mar 14, 2023
Closed
@lemeurherve lemeurherve mentioned this issue Dec 12, 2023
Open
@lemeurherve
Copy link
Member Author

We'll use https://github.com/Azure/azure-storage-azcopy.

@lemeurherve lemeurherve self-assigned this Dec 13, 2023
@lemeurherve
Copy link
Member Author

Current references to blobxfer (excluding some updatecli manifests and co):

Corresponding storage accounts:

  • prodpluginsite, File Share: pluginsite
  • prodjenkinsio, File Share: jenkinsio (and also cnjenkinsio & zhjenkinsio)
  • prodjavadoc, File Share: javadoc
  • prodjenkinsreleases, File Shares: mirrorbits & website

To replace blobxfer by azcopy for manipulating these files shares, SAS tokens need to be generated and stored as credentials (blobxfer is using the access key of the storage accounts, much less grain fined).
To do so, these storage accounts and file shares have to be imported as code in https://github.com/jenkins-infra/azure first.

The plan is to:

Note: if time permits, we should also replace az storage by azcopy in infra.publishReports: https://github.com/jenkins-infra/pipeline-library/blob/350e1b561b633ae204265ce89bca20edfce3c02f/vars/publishReports.groovy#L55-L59

@timja
Copy link
Member

timja commented Dec 13, 2023 •
edited

To replace blobxfer by azcopy for manipulating these files shares, SAS tokens need to be generated and stored as credentials (blobxfer is using the access key of the storage accounts, much less grain fined).
To do so, these storage accounts and file shares have to be imported as code in jenkins-infra/azure first.

Ideally a service principal / managed identity / workload identity should be used instead.
SaS tokens are irrevocable and are worse than account keys in some ways

@lemeurherve
Copy link
Member Author

Oh... I'll look into that instead then, thanks for the info @timja

@dduportal
Copy link
Contributor

This plan looks good and exhaustive. The only (blocking) point will be the credential: we need to document what is the exact kind of token required and what is the process to revoke it

@lemeurherve
Copy link
Member Author

Ideally a service principal / managed identity / workload identity should be used instead. SaS tokens are irrevocable and are worse than account keys in some ways

Unfortunately azcopy supports only SAS token for File Share: https://learn.microsoft.com/en-gb/azure/storage/common/storage-use-azcopy-v10#authorize-azcopy

According to https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#manually-rotate-access-keys and https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas#revoke-a-sas, it's possible to revoke SAS tokens by rotating or regenerating the storage account access key.

@timja
Copy link
Member

timja commented Dec 14, 2023

Depending on how the file share is being used you can still use a service principal to generate a SaS on demand:
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-user-delegation-sas-create-cli

This was referenced Dec 19, 2023
Merged
Merged
lemeurherve added a commit to jenkins-infra/azure that referenced this issue Dec 20, 2023
@lemeurherve
feat: allow trusted.ci.jenkins.io to manipulate storage (#557)
33647ff 
This PR uses jenkins-infra/shared-tools#131 to
allow trusted.ci.jenkins.io to manipulate the File Share content of
updates.jenkins.io

Ref:
-
jenkins-infra/helpdesk#3414 (comment)
@lemeurherve lemeurherve mentioned this issue Dec 20, 2023
Merged
This was referenced Mar 28, 2024
Merged
Merged
@dduportal
Copy link
Contributor

PRs for cleanup of blobxfer on the pkg VM:

@dduportal
Copy link
Contributor

PRs for cleanup of blobxfer on the pkg VM:

* [cleanup(pkgrepo) remove last remnants of blobxfer jenkins-infra#3354](https://github.com/jenkins-infra/jenkins-infra/pull/3354)

* [cleanup: remove last remnants of blobxfer mirror-scripts#25](https://github.com/jenkins-infra/mirror-scripts/pull/25)

Deployed with success and ran the cleanup process. Watching update center and sync.sh

@dduportal
Copy link
Contributor

PRs for cleanup of blobxfer on the pkg VM:

* [cleanup(pkgrepo) remove last remnants of blobxfer jenkins-infra#3354](https://github.com/jenkins-infra/jenkins-infra/pull/3354)

* [cleanup: remove last remnants of blobxfer mirror-scripts#25](https://github.com/jenkins-infra/mirror-scripts/pull/25)

Deployed with success and ran the cleanup process. Watching update center and sync.sh

Looks good!

@dduportal dduportal mentioned this issue Mar 28, 2024
Merged
@dduportal
Copy link
Contributor

Managing mirrorbrain (crontab, user, ssh keys, etc.) user and the .azure-storage-env file in jenkins-infra/jenkins-infra#3357

@dduportal
Copy link
Contributor

Update: jenkins-infra/jenkins-infra#3357 (comment) was successfully deployed.

@lemeurherve I'm handing over to you for the SA token cleanup you mentioned as I have no idea what to cleanup and this nis the last step before closing this issue.

This was referenced Mar 29, 2024
Closed
Merged
Merged
This was referenced Apr 2, 2024
Merged
Merged
This was referenced Apr 3, 2024
Merged
Merged
@lemeurherve
Copy link
Member Author

lemeurherve commented Apr 3, 2024 •
edited

@lemeurherve I'm handing over to you for the SA token cleanup you mentioned as I have no idea what to cleanup and this nis the last step before closing this issue.

Here are the last cleanup tasks remaining before we can close this issue:

This was referenced Apr 4, 2024
Merged
Merged
Merged
lemeurherve added a commit to jenkins-infra/azure that referenced this issue Apr 4, 2024
@lemeurherve
cleanup(get.jenkins.io): remove long-lived SAS token and related outp…
1f0274f 
…uts (#654)

This PR removes `mirrorbits` file share long-lived SAS token and related
outputs, replaced by the use of a storage key stored in jenkins-infra
hieradata & charts-secrets.

Verification procedure after merging this PR: ensure update_center job
still passes on trusted.ci.jenkins.io (file share used in sync.sh &
sync-recent-releases.sh scripts)

Ref:
-
jenkins-infra/helpdesk#3414 (comment)
lemeurherve added a commit to jenkins-infra/azure that referenced this issue Apr 4, 2024
@lemeurherve
cleanup(contributors.jenkins.io): remove `azurerm_storage_share.contr…
5447ff4 
…ibutors_jenkins_io` output (#655)

This PR removes this output, superseded by the use of
https://github.com/jenkins-infra/pipeline-library/blob/master/resources/get-fileshare-signed-url.sh
to get the file share URL including a SAS token.

Verification procedure after merging this PR: ensure
https://infra.ci.jenkins.io/job/website-jobs/job/contributor-spotlight/
job passes.

Ref:
-
jenkins-infra/helpdesk#3414 (comment)
@lemeurherve lemeurherve mentioned this issue Apr 4, 2024
Merged
@lemeurherve
Copy link
Member Author

blobxfer completely replaced by azcopy, cleanup done, all concerned jobs green, closing this issue.

@dduportal dduportal mentioned this issue Apr 16, 2024
Closed
@lemeurherve lemeurherve mentioned this issue May 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lemeurherve lemeurherve

Labels
blobxfer get.jenkins.io javadoc jenkins.io site:archives.jenkins.io Related to this infrastructure service trusted.ci.jenkins.io update-center
Projects
None yet
Milestone
infra-team-sync-2024-04-09
Development

No branches or pull requests
4 participants
@dduportal @timja @lemeurherve @smerle33