feat(pkg) add jenkins-infra/mirrorscripts under Puppet management #3358

dduportal · 2024-03-29T09:04:34Z

Follow up of jenkins-infra/helpdesk#3414

This PR is the first step of #3352.

It adds the set of "mirror scripts" used on the pkg VM (pkg.origin.jenkins.io and updates.jenkins.io) under Puppet management but without any templated content (subsequent PR):

The repository jenkins-infra/mirror-scripts was imported into dist/profile/templates/mirror-scripts with all its history using the following command (note: 7184a97f1a2591684f6b5b32abaf7ff5efb1dfd3 being the HEAD of its main branch):
```
git subtree add -P dist/profile/templates/mirror-scripts https://github.com/jenkins-infra/mirror-scripts 7184a97f1a2591684f6b5b32abaf7ff5efb1dfd3
```
- Note that any git log command in the future will see the imported reference files at the root of jenkins-infra/jenkins-infra so might need addition options. But a subsequent PR will have the script renamed with a .erb extension.
There is only one commit to be review: the last one e8a6aebf69ae0898f44cdabf3a8e052e3e5d0302 which has the puppet management code.
- Note that the file content uses the template() function but no templated content is available yet!
- The .gitignore and the README.md files have been removed from mirror-scripts

The following workflow is expected once this PR is approved (or if a "force merge" is triggered) - tested on a local vagrant VM

before merging:
- Run the puppet agent on the VM one last time to ensure the "noop" diff won't be polluted by the usual Jenkins keys regular changes (will be fixed on another PR)
- Disabling puppet agent before on the pkg VM
Merge the PR here
Run a puppet agent "noop" and make sure that only the changes here are expected, then apply
-Note: We only expect the 6 symlinks to be replaced by files
Compare the content of the 6 new files with the content of their counterparts in /srv/releases/mirror-scripts/. They should be the same
If OK, then delete the directory /srv/releases/mirror-scripts and the /srv/releases/.git* config file (if any)
Restart the pupept agent and check there are no changes

I'm aware that some of the scripts are in jenkins-infra/jenkins-infra but the puppet agent has been disabled for ages and this is an easy way to ensure we don't lose changes

Handle the case when there are no recent releases

Co-authored-by: mirrorbrain Puppet-managed User <mirrorbrain@ip-172-31-33-96.ec2.internal>

Update README.md

Server is down and cannot receive the files.

…pload Suspend archives upload

This reverts commit b8d247e.

…d-archives-upload Revert "Suspend archives upload"

Server is down and cannot receive the files. See jenkins-infra/mirror-scripts#5 for an example and jenkins-infra/mirror-scripts#6 for the change that reverted it when archives.jenkins.io was again available.

…archives Temporarily disable upload to archives.jenkins-ci.org

…e-upload-to-archives Revert "Temporarily disable upload to archives.jenkins-ci.org"

sync-recent-releases must copy directory content

Ref: - jenkins-infra/helpdesk#2967 - jenkins-infra/helpdesk#2888

in order to solve dependencies conflict

fix(sync recent releases): adding a venv python for blobxfer

…blobxfer CLI Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

fix(batch upload,sync) follow up of 503b7ac to use a virtual env for blobxfer CLI

…PATH when running from a crontab Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

…or azcopy Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

cleanup: remove last remnants of blobxfer

…591684f6b5b32abaf7ff5efb1dfd3' git-subtree-dir: dist/profile/templates/mirror-scripts git-subtree-mainline: 9d388e8 git-subtree-split: 7184a97

…-it (content not templatized yet) Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

lemeurherve · 2024-03-29T12:42:53Z

WDYT of rebasing some of the recent commits?

dduportal · 2024-03-29T12:44:51Z

WDYT of rebasing some of the recent commits?

I'm not sure to understand: what is the problem it would solve? Not sure how to do it either?

lemeurherve · 2024-03-29T13:13:05Z

Sorry, by rebasing I meant squashing commits so the git (recent) history would be a little bit cleaner and easier to navigate if needed.

MarkEWaite · 2024-03-29T14:09:31Z

Sorry, by rebasing I meant squashing commits so the git (recent) history would be a little bit cleaner and easier to navigate if needed.

I'm opposed to rebase or squash of changes on the primary branch of a repository because it requires others who have forked or cloned the repository to perform additional actions to replace the history that is being rewritten. That is especially problematic when git clone is used to populate a tools collection onto a machine. I do that with agents in my private Jenkins controller and it would be a terrible experience if I changed the existing history of the primary branch.

I don't object to squash or rebase during a merge operation. That tends to keep the history of the primary branch more tidy. However, once something is on the primary branch, a rewrite of history creates pain for others.

lemeurherve · 2024-03-29T14:24:33Z

Sorry, by rebasing I meant squashing commits so the git (recent) history would be a little bit cleaner and easier to navigate if needed.

I'm opposed to rebase or squash of changes on the primary branch of a repository because it requires others who have forked or cloned the repository to perform additional actions to replace the history that is being rewritten. That is especially problematic when git clone is used to populate a tools collection onto a machine. I do that with agents in my private Jenkins controller and it would be a terrible experience if I changed the existing history of the primary branch.

I don't object to squash or rebase during a merge operation. That tends to keep the history of the primary branch more tidy. However, once something is on the primary branch, a rewrite of history creates pain for others.

As these commits aren't in this repo yet I'm proposing to take the opportunity of this PR to squash some of these.
No one should be impacted by this as there is no rebase or squash of any commit of the primary branch, only on commits of this pull request.

dduportal · 2024-03-29T14:32:47Z

Sorry, by rebasing I meant squashing commits so the git (recent) history would be a little bit cleaner and easier to navigate if needed.

I'm opposed to rebase or squash of changes on the primary branch of a repository because it requires others who have forked or cloned the repository to perform additional actions to replace the history that is being rewritten. That is especially problematic when git clone is used to populate a tools collection onto a machine. I do that with agents in my private Jenkins controller and it would be a terrible experience if I changed the existing history of the primary branch.
I don't object to squash or rebase during a merge operation. That tends to keep the history of the primary branch more tidy. However, once something is on the primary branch, a rewrite of history creates pain for others.

As these commits aren't in this repo yet I'm proposing to take the opportunity of this PR to squash some of these. No one should be impacted by this as there is no rebase or squash of any commit of the primary branch, only on commits of this pull request.

The goal of importing the history is to allow navigating through the git CLI instead of having to search GitHub (https://github.com/jenkins-infra/mirror-scripts/commits/main/). It means the history should be kept to map to the existing (almost archived) repository otherwise I don't see the need.

Rewriting the imported history has an impact: it will be hard to map a given change to the associated history in https://github.com/jenkins-infra/mirror-scripts.

Besides, what would the criteria to decide to rebase or not? Who would have to do this? It looks like a superficial change and that means spending hours getting a review + consensus on each squashed commit. I don't see the problem being solved by doing this, but only the waste of time. Could you clarify your request in that matter?

The set of "mirror scripts" imported in jenkins-infra#3358 have their content templatized to map to puppet hieradata configuration used by profiles. Note: the profile `updatesite` has been updated (default param. value move to hieradata) to allow other profiles to lookup the value (`updatesite` and `pkgrepo`` are still highly coupled on the same VM). Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

dduportal · 2024-03-29T15:25:09Z

Starting a pairing session with @smerle33 to land this PR and process forward.

We'll merge with the option "Merge commit" to avoid polluting the main branch's history. The git log will still be able to search the imported history through the ref-log, but we won't have what appear to be an annoyance regarding the collection of commits.

smerle33

LGTM 🚀

dduportal · 2024-03-29T15:57:05Z

Deployment went well (following the written procedure).

The set of "mirror scripts" imported in jenkins-infra#3358 have their content templatized to map to puppet hieradata configuration used by profiles. Note: the profile `updatesite` has been updated (default param. value move to hieradata) to allow other profiles to lookup the value (`updatesite` and `pkgrepo`` are still highly coupled on the same VM). Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

* chore(pkg) templatize mirror-scripts on the 'pkgrepo' profile The set of "mirror scripts" imported in #3358 have their content templatized to map to puppet hieradata configuration used by profiles. Note: the profile `updatesite` has been updated (default param. value move to hieradata) to allow other profiles to lookup the value (`updatesite` and `pkgrepo`` are still highly coupled on the same VM). Signed-off-by: Damien Duportal <damien.duportal@gmail.com> * Update dist/profile/manifests/pkgrepo.pp * Update dist/profile/manifests/pkgrepo.pp * code review fixup Signed-off-by: Damien Duportal <damien.duportal@gmail.com> * code review fixup Signed-off-by: Damien Duportal <damien.duportal@gmail.com> * code review fixup Signed-off-by: Damien Duportal <damien.duportal@gmail.com> * code review fixup Signed-off-by: Damien Duportal <damien.duportal@gmail.com> * Update spec/classes/profile/pkgrepo_spec.rb --------- Signed-off-by: Damien Duportal <damien.duportal@gmail.com> Co-authored-by: Stéphane MERLE <95630726+smerle33@users.noreply.github.com>

mirrorbrain Puppet-managed User and others added 30 commits August 21, 2020 19:19

Initial commit

c777856

Copy scripts from pkg server

1bfe81d

I'm aware that some of the scripts are in jenkins-infra/jenkins-infra but the puppet agent has been disabled for ages and this is an easy way to ensure we don't lose changes

Remove sync-basic

1482b67

Merge pull request jenkins-infra#1 from jenkins-infra/initial-push

b535959

Handle the case when there are no recent releases

68d5bd4

Merge pull request jenkins-infra#2 from jenkins-infra/no-recent-releases

9a807ed

Handle the case when there are no recent releases

Sync recent releases to archives and OSU immediately (jenkins-infra#3)

e694f3f

Co-authored-by: mirrorbrain Puppet-managed User <mirrorbrain@ip-172-31-33-96.ec2.internal>

Update README.md

5026ead

Merge pull request jenkins-infra#4 from jenkins-infra/olblak-patch-1

a7123d8

Update README.md

Use absolute path to blobxfer command

53ada92

Temporarily disable upload to archives.jenkins-ci.org

b8d247e

Server is down and cannot receive the files.

Merge pull request jenkins-infra#5 from MarkEWaite/suspend-archives-u…

c26a198

…pload Suspend archives upload

Revert "Temporarily disable upload to archives.jenkins-ci.org"

4b0bb8a

This reverts commit b8d247e.

Merge pull request jenkins-infra#6 from jenkins-infra/revert-5-suspen…

97eb6c4

…d-archives-upload Revert "Suspend archives upload"

Temporarily disable upload to archives.jenkins-ci.org

c4a789d

Server is down and cannot receive the files. See jenkins-infra/mirror-scripts#5 for an example and jenkins-infra/mirror-scripts#6 for the change that reverted it when archives.jenkins.io was again available.

Merge pull request jenkins-infra#7 from MarkEWaite/disable-upload-to-…

37b6f09

…archives Temporarily disable upload to archives.jenkins-ci.org

Revert "Temporarily disable upload to archives.jenkins-ci.org"

bd156b4

Merge pull request jenkins-infra#8 from jenkins-infra/revert-7-disabl…

f5bdb44

…e-upload-to-archives Revert "Temporarily disable upload to archives.jenkins-ci.org"

sync-recent-releases copies directory content

9a1dbc1

Merge pull request jenkins-infra#9 from jenkins-infra/olblak-patch-1

9b4d4c6

sync-recent-releases must copy directory content

Update command to trigger archives sync

947a0e4

fix(remove mirrorbrain scan command)

bd6dcfc

Ref: - jenkins-infra/helpdesk#2967 - jenkins-infra/helpdesk#2888

fix(blobxfer): adding a venv python

503b7ac

in order to solve dependencies conflict

add requirements.txt

ac1f0a6

Merge pull request jenkins-infra#10 from jenkins-infra/smerle33-patch-1

f7755a6

fix(sync recent releases): adding a venv python for blobxfer

fix(batch upload,sync) follow up of 503b7ac to use a virtual env for …

195f598

…blobxfer CLI Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

Merge pull request jenkins-infra#11 from dduportal/fix/blobxfer/venv

8ee32b7

fix(batch upload,sync) follow up of 503b7ac to use a virtual env for blobxfer CLI

Create .gitignore

13a13d3

Update requirements.txt

49abc7c

fix(sync.sh) use blobxfer from path (venv) instead of abs.path

bebd767

dduportal added 6 commits March 27, 2024 15:28

hotfix(sync.sh) ensure get-fileshare-signed-url.sh can be found from …

674697a

…PATH when running from a crontab Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

hotfix(sync-recent-release) revert '--overwrite=ifSourceNewer' flag f…

a99b89c

…or azcopy Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

cleanup: remove last remnants of blobxfer

2e370d4

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

Merge pull request jenkins-infra#25 from jenkins-infra/cleanup/blobxfer

7184a97

cleanup: remove last remnants of blobxfer

Add 'dist/profile/templates/mirror-scripts/' from commit '7184a97f1a2…

5c1e473

…591684f6b5b32abaf7ff5efb1dfd3' git-subtree-dir: dist/profile/templates/mirror-scripts git-subtree-mainline: 9d388e8 git-subtree-split: 7184a97

feat(profile::pkgrepo) ensure mirror-scripts are managed by Puppet as…

e8a6aeb

…-it (content not templatized yet) Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

dduportal changed the title ~~Feat/pkg/add mirrorscripts~~ feat(pkg) add jenkins-infra/mirrorscripts under Puppet management Mar 29, 2024

dduportal marked this pull request as ready for review March 29, 2024 12:07

dduportal requested a review from a team as a code owner March 29, 2024 12:07

dduportal requested review from MarkEWaite, timja, lemeurherve and smerle33 March 29, 2024 12:07

This was referenced Mar 29, 2024

chore(pkg) templatize mirror-scripts on the 'pkgrepo' profile #3359

Merged

[pkg.origin.jenkins.io] Manage the jenkins-infra/mirror-scripts scripts with Puppet #3352

Closed

dduportal assigned smerle33 Mar 29, 2024

smerle33 approved these changes Mar 29, 2024

View reviewed changes

smerle33 merged commit 79bde8f into jenkins-infra:production Mar 29, 2024
2 checks passed

dduportal deleted the feat/pkg/add-mirrorscripts branch March 29, 2024 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pkg) add jenkins-infra/mirrorscripts under Puppet management #3358

feat(pkg) add jenkins-infra/mirrorscripts under Puppet management #3358

dduportal commented Mar 29, 2024 •

edited

lemeurherve commented Mar 29, 2024

dduportal commented Mar 29, 2024

lemeurherve commented Mar 29, 2024

MarkEWaite commented Mar 29, 2024

lemeurherve commented Mar 29, 2024 •

edited

dduportal commented Mar 29, 2024

dduportal commented Mar 29, 2024

smerle33 left a comment

dduportal commented Mar 29, 2024

feat(pkg) add jenkins-infra/mirrorscripts under Puppet management #3358

feat(pkg) add jenkins-infra/mirrorscripts under Puppet management #3358

Conversation

dduportal commented Mar 29, 2024 • edited

lemeurherve commented Mar 29, 2024

dduportal commented Mar 29, 2024

lemeurherve commented Mar 29, 2024

MarkEWaite commented Mar 29, 2024

lemeurherve commented Mar 29, 2024 • edited

dduportal commented Mar 29, 2024

dduportal commented Mar 29, 2024

smerle33 left a comment

Choose a reason for hiding this comment

dduportal commented Mar 29, 2024

dduportal commented Mar 29, 2024 •

edited

lemeurherve commented Mar 29, 2024 •

edited