[INFRA-3100] Migrate updates.jenkins.io to another Cloud

[jenkins-infra-bot]

Why

Read the EPIC (aws cost decrease: https://github.com//issues/2646)

What

• updates.jenkins.io serves the JSON of the update center, but it is not behind a CDN (need to be updated regularly) causing a lot of outbound data transfer (2.4 Mb for a JSON download, esitmation around 300 Gb of data sent daily, estimation around 10 Tb per month which costs some $$$.

• The VM pkg.origin.jenkins.io is managing multiple services and separting them could be a great help.

How

Different paths can be taken here: to be discussed in infra meeting + validated by the board as it's an important service.

• Migrating the service to Oracle Cloud:
  • the first 10 Tb of outbound data is free, and if the worst case, 50 Tb of outbound would cost less than 3k $ per month - https://www.oracle.com/be/cloud/networking/networking-pricing.html
  • ARM machines: cheaper and better performance for a simple webserver + SSH + data
  • Block storage: close to archives.jenkins.io
  • Risk: Oracle sponsoring program is only 1 year, and is not easily secured for paiment as other account

• Migrating to Azure
  • Easily in AKS: easier management
  • Safer paiment process
  • Risk: cost of outbound to be evaluated

---

Originally reported by dduportal, imported from: Migrate updates.ci.jenkins.io to another Cloud

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[note]

• Check that Claim that updates.jenkins.io certificate is sometimes expiring #3017 is not happening once migration is finished.
• The error curl: (16) Error in the HTTP2 framing layer should not happen anymore with the latest Apache versions

[dduportal]

Requires setting up the Oracle Terraform project, like #2682

Todo list:

• Specify a new infra (VM + data storage of the same size as the actual pkg.origin.jenkins.io machine + any other Oracle infra requirements). Implies checking the pricing to get an overall idea
• Once VM is created, add it to puppet management: new node, with the role / profiles associated to "update.jenkins.io"
• Rsync the data one time
• Update the jenkins-infra/update-center2 's associated job in trusted.ci to update both "pkg.origin.jenkins.io" VM and the new one (⚠️ inline pipeline, don't search for a Jenkinsfile as code)
• Validate the new instance (including checking with Daniel, Tim and other contributors)
• Communicate about the upcoming change
• As proposed by Stephane, use a round-robin DNS record value to split traffic between old/new during some time and check for error

[dduportal]

Blocked by #2973

[smerle33]

actual machine size is :
32Gb RAM
8cpu
1,2Tb data disk (372Gb free)

about half of the power is used currently (checked with the local SAR probe)

[smerle33]

Infra to specify :

• 1 VM (similar to archive.jenkins.io)
• 1 network ("mirrors") + 1 subnet ("20210630-1531")
• 1 volume 1,2Tb
• 1 set of security groups to restrict network access
• 1 ssh key pair

[smerle33]

VM specifications :

• 4vCPU/16Gb RAM : half of actual machine
• proposal to use ARM like archive.jenkins.io --> 4ocpu (VM Type Standard A1 Flex)
• Image Ubuntu 20.04 (FULL [non minimal]) --> upgrade from ubuntu 18.04 for actual machine
  • option for 22.04 but need testing for puppet agent (+ openssl v3)

[dduportal]

Update:

• WiP on using a distinct filesystem (with only .htaccess and directory struct.) for httpd on azure.updates.jenkins.io:
  • Removed httpd deployment from the mirrorbits-parent release: chore(updates.jenkins.io): disable httpd service in mirrorbits-parent release kubernetes-management#5186
  • Tentative to use the existing httpd chart code to specify a new PV/PVC: feat(updates.jenkins.io): dedicated File Share for the httpd service kubernetes-management#5184. Failed and rollbacked in Revert "feat(updates.jenkins.io): dedicated File Share for the httpd service" kubernetes-management#5190.
  • Sync with @hervelemeur and we were able to pair-diagnose that the httpd chart need an enhancement to support the CSI PV/PVC syntax + secret ref. => next step taken by @dduportal

[dduportal]

Update:

• WiP on using a distinct filesystem (with only .htaccess and directory struct.) for httpd on azure.updates.jenkins.io:

  • Removed httpd deployment from the mirrorbits-parent release: chore(updates.jenkins.io): disable httpd service in mirrorbits-parent release kubernetes-management#5186
  • Tentative to use the existing httpd chart code to specify a new PV/PVC: feat(updates.jenkins.io): dedicated File Share for the httpd service kubernetes-management#5184. Failed and rollbacked in Revert "feat(updates.jenkins.io): dedicated File Share for the httpd service" kubernetes-management#5190.
  • Sync with @hervelemeur and we were able to pair-diagnose that the httpd chart need an enhancement to support the CSI PV/PVC syntax + secret ref. => next step taken by @dduportal

Update:

• Only a set of config (and secret) were required. The httpd service is now deployed and we can access it at azure.updates.jenkins.io
• However there still are (minor- issues regarding the TLS termination). Incoming configuration fix!

[dduportal]

* However there still are (minor- issues regarding the TLS termination). Incoming configuration fix!

Update:

• TLS issues fixed: both azure.updates.jenkins.io and mirrors.updates.jenkins.io have a proper TLS termination managed by Let's Encrypt!
• azure.updates.jenkins.io is now proudly serving ... nothing... but with Apache (httpd) ✅

[dduportal]

Update:

The storage account for updates-jenkins-io started to cost a lot on our Azure main subscription:

(Note: C1 Cache is the managed Redis service, but any LRS .* or .* Operations Meter is related to the storage account)

As you can see on the above screenshot, as soon as we enabled the update_center update of JSON index on the new UC (in parallel of current UC), the costs on the file share increased immediately.

=> We must switch to a Premium file storage like we did for get.jenkins.io. Expected new costs should be (source https://azure.microsoft.com/en-us/pricing/details/storage/files/):

• Data storage should increase from $0.09 monthly to $0,50 $32 (2 FS at 100 Gb each at $0.16 per Gb).
• Transactions should decrease from ~$380 monthly to $0 (yes, zero dollar)

=> Total should change from ~$391 monthly to $32 monthly (10x decrease).

Plan:

• Opt out update_center from the azure + R2 updates (keep updating the pkg VM) as we'll destroy the current storage account (ref. https://github.com/jenkins-infra/update-center2/blob/72f646e39f60864c5ad6e917e4e7423ac5252b1b/site/publish.sh#L4)
• Disable updates-jenkins-io 's publick8s helmfile releases and ensure PVC/PV are removed (and pods not up)
  • hotfix(publick8s) stop managing updates-jenkins-io releases until storage is migrated to premium kubernetes-management#5199
  • kubectl delete ns updates-jenkins-io
  • kubectl delete pv updates-jenkins-io-mirrorbits-parent
  • kubectl delete pv updates-jenkins-io-httpd
• Upgrade the current storage account to Premium (requires destroy + re-create)
  • feat!(publick8s/updates.jenkins.io) switch Storage Account to Premium to decrease costs (BREAKING) azure#682
  • fix(updates.jenkins.io) set FS to minimum size for each azure#683 (fixup to provision min. 100 Gb per file share)
• Re-install updates-jenkins-io 's publick8s helmfile releases with new PVC/PV using the new premium storage
  • feat(publick8s/updates.jenkins.io) install with premium storage kubernetes-management#5200 (review)
• Update credentials and setup on trusted.ci on crawler and test it (we want the deployment to happen on the new azure storage)
  • https://github.com/jenkins-infra/crawler/blob/77a44bb9f3bdf47a217804463177a1b2d74e04c0/Jenkinsfile#L74C37-L74C92
  • Forgot about the kubeconfig. Added it in our SOPS repository (https://github.com/jenkins-infra/charts-secrets/commit/fd1f2ac12302dd42a172a429d7a464e035556f22) and updated on trusted.ci
• Opt in again update_center and make sure eveyrthing work as expected
  • crawler is green
  • update_center is green
  • https://azure.updates.jenkins.io/ => LGTM (empty webserver)
  • https://mirrors.updates.jenkins.io/current/update-center.actual.json?mirrorlist => LGTM (mirrorbit status page and redirect to cloudflare)

@hervelemeur I'm passing back to you for the www-* updates and sanity checks

[dduportal]

Update:

• The PR feat(publish.sh): use dedicated source folder for httpd service including only .htaccess files update-center2#777 aims at filling both file shares using the proper content (no htaccess on mirrorbits, no content to be served accidentallt on httpd). It introduces the need to load 2 credentials in parallel which will rely on env file with expected naming (to avoid dereferencing variables)
• The new credential has been created, is documented in Terraform and available on trusted.ci's test job. It is a ZIP file managed as a Jenkins File credential (temporarly unzipped in a temp directory)
  • chore(trusted.ci.jenkins.io): ouput to generate a zip of update center env files to access file shares azure#690
  • fix(trusted.ci.jenkins.io): correct name and output for the httpd file share service principal writer azure#692
• Handover from @hervelemeur to @dduportal last Friday:
  • a bit of codestyle (see PR)
  • Agreed on the next steps (see below)

Next steps:

• Finish update_center2's PR code cleanup and tests and merge it once approved => @dduportal with approval by @daniel-beck when ready
• Start functional tests => @dduportal and @smerle33 (pair?)
• Start performance stress tests => @dduportal and @smerle33 (pair?)
• Create a second Cloudflare mirror in US-east => @smerle33 with reviews by @dduportal