[infra.ci.jenkins.io] Azure AD Password expired for terraform-production (jenkins-infra/azure)

[dduportal]

Service(s)

Azure, infra.ci.jenkins.io

Summary

The credentials used by infra.ci.jenkins.io to manage the Terraform project jenkins-infra/azure are expired: any job quickly fail on early Terraform phases when accessing the shared state with the following error:

Error: Failed to get existing workspaces: Error retrieving keys for Storage Account "<redacted>": autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys for app '<redacted>' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds. Trace ID: <redacted> Correlation ID: <redacted> Timestamp: 2024-04-16 10:37:34Z","error_codes":[7000222],"timestamp":"2024-04-16 10:37:34Z","trace_id":"<redacted>","correlation_id":"<redacted>","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

Reproduction steps

No response

[dduportal]

Rotated credentials with the following change (private link): https://github.com/jenkins-infra/terraform-states/commit/bbab9fe5e97ce12d1bdcc9be5581941515e7dbe1

It involved:

• Updating the module: as it changed from using Azure SP password to AzureAD Application password (as per @lemeurherve 's findings in Check if we could replace blobxfer by azcopy #3414 )
• Adding a 3-month expiration date (TODO: calendar update!)

=> we'll have to apply this change (rotation + expiry) to all the other projects in https://github.com/jenkins-infra/terraform-states

[dduportal]

Update:

• Updated the credentials in infra.ci.jenkins.io vault: https://github.com/jenkins-infra/charts-secrets/commit/821bead348a5486c9b830389163ef340f50d694a
  • As usual, it required a successful run of kubernetes-management
• The job Terraform Azure is now working as expected: e.g. it performs all the steps including terraform apply without any permission or credential errors
  • Note: [infra.ci.jenkins.io] Terraform Azure job fails with Total node count 56 requires 71680 ports but only 64000 ports are available given 1 outbound public IPs #4046 has been open with the new Azure error we have when applying. This error is unrelated to the Azure credentials

[dduportal]

Next steps before closing:

• As https://github.com/jenkins-infra/terraform-states/commit/bbab9fe5e97ce12d1bdcc9be5581941515e7dbe1 changed the terraform module used to set up the Terraform Shared Backends (for states), we have to apply the change to all project (and deploy the updated secret backend-config files to infra.ci)
• The project jenkins-infra/azure-net also defines, like jenkins-infra/azure, a couple of Azure Service Principal: they need to be updated in the same way (with explicit expiration date)
• We have to add the expiration dates to our shared calendar: even if notifications are not shared, we should have it as a strict minimum
• Open an issue about how to run updatecli on the (private!) repository jenkins-infra/terraform-states to update the expiration dates + providers

[dduportal]

• As jenkins-infra/terraform-states@bbab9fe changed the terraform module used to set up the Terraform Shared Backends (for states), we have to apply the change to all project (and deploy the updated secret backend-config files to infra.ci)
• The project jenkins-infra/azure-net also defines, like jenkins-infra/azure, a couple of Azure Service Principal: they need to be updated in the same way (with explicit expiration date)

Update: jenkins-infra/azure-net has been updated:

• https://github.com/jenkins-infra/terraform-states/commit/79e9b812b99bc2ad3d9770f31905fc657e53e9f9
• https://github.com/jenkins-infra/charts-secrets/commit/1791d5ea373d9e7afa505d71f463457840f6284b
• infra.ci credentials "forced" reload:
  • kubernetes-management run with success (to apply new credentials into the pod secrets)
  • JCasc reload (to tell the Jenkins controller to re-eavluate the pod secrets)
  • Build Queue cleaned up with the Jenkins.instance.queue.clear() instruction run in the Groovy Script Console
• Verified build is green with a rebuild of Generate a new Azure Service Principal password with expiration date 2024-07-17T00:00:00Z on cert.ci.jenkins.io azure-net#218

[dduportal]

• As jenkins-infra/terraform-states@bbab9fe changed the terraform module used to set up the Terraform Shared Backends (for states), we have to apply the change to all project (and deploy the updated secret backend-config files to infra.ci)
• The project jenkins-infra/azure-net also defines, like jenkins-infra/azure, a couple of Azure Service Principal: they need to be updated in the same way (with explicit expiration date)

The other projects have been updated (jenkins-infra/aws, jenkins-infra/cloudflare, jenkins-infra/digitalocean and jenkins-infra/fastly). Note that jenkins-infra/datadog has NOT been updated: it appears that its state is NOT managed in jenkins-infra/terraform-states (most probably never migrated from legacy state).

Same method has been used:

• Distinct commits on terraform-states (along with chore updates):
  • https://github.com/jenkins-infra/terraform-states/commit/59d4bbaa6a3b5d763c7e4d23168f41de12d46562
  • https://github.com/jenkins-infra/terraform-states/commit/159787bdf349095062da3d0f37bf0245afd2890a
  • https://github.com/jenkins-infra/terraform-states/commit/3d8f885396c1390d31e431c870f0171ee6af52e6
  • https://github.com/jenkins-infra/terraform-states/commit/e688bc525e837941a1102ebbef535d20a84ea6f0
• Distinct commits on chart-secrets:
  • https://github.com/jenkins-infra/charts-secrets/commit/d5a7b994bad6fc16b21a5ab817ba187fc3fb86c8
  • https://github.com/jenkins-infra/charts-secrets/commit/dad6e8103d0df445c63baae136b6f449d6e96447
  • https://github.com/jenkins-infra/charts-secrets/commit/a09742d213ba1c0a08fc092ed9c492b178224ac5
  • https://github.com/jenkins-infra/charts-secrets/commit/cae4e0324e522f3d88fc43db0ea9a743ea25dbbe
• infra.ci reload (with all the sub tasks)
• Validated that the 4 projects are still building (main branch or PR builds) with new credentials

[dduportal]

We have to add the expiration dates to our shared calendar: even if notifications are not shared, we should have it as a strict minimum

Update: calendar events added. it won't 100% ensure we catch it before expiration but it is a start!

[dduportal]

Open an issue about how to run updatecli on the (private!) repository jenkins-infra/terraform-states to update the expiration dates + providers

Issue opened: #4051