Test KKP Upgrade - Test Release 2.25

[embik]

As usual (see #12799 for previous ticket), we need to run a couple of upgrade tests to make sure we don't run into problems with KKP 2.24 being upgraded to KKP 2.25.

Please add your things to test in comments below, so I can add them to the things to be validated during and after a KKP upgrade.

To Be Validated

Beta Give feedback

1. Create clusters in various configurations (versions, CNI, addons) in 2.24 and observe their upgrade behaviour
2. Use app catalog feature, install apps in 2.24 and upgrade KKP installation to 2.25 to observe app upgrades
3. Deploy MLA stacks and observe their upgrade behaviour

Found Issues

Beta Give feedback

1. IAP upgrade fails with no such file or directory for test.sh symlink #13142
   kind/bug sig/app-management +2
   
2. All application installations installed from default app catalog were deleted on upgrade to v2.25.0-beta.2 #13143
   kind/bug sig/app-management +2
   
3. AWS CCM crash loops after upgrade to v2.25.0-beta.2 and cloud spec cannot be upgraded #13144
   kind/bug sig/cluster-management +2
   
4. Application catalog is missing icons after upgrading from KKP v2.24.4 #13145
   kind/bug sig/app-management +2
   
5. usercluster-controller-manager crash looping after disabling CSI and upgrading to newer Kubernetes version #13146
   kind/bug sig/cluster-management +2
   

[SimonTheLeg]

Update: Since we are going with a non-breaking version, this no longer needs to be verified
To be validated: With the change of the values and defaultValues to a string in #13053, we need to verify that this does not brick the upgrade process

[archups]

To be validated: With the change of the helm chart version for default applications in #13058, we need to verify that this does not brick the upgrade process.

[wurbanski]

to be validated: MLA upgrades:

• loki upgrade (changes to upgrade user-mla loki to v2.9.2 via helm chart v0.78.0 #12967)
• grafana upgrade ([release/v2.22] update version matrix for EKS/AKS (#12962) #12965
• minio upgrade (including fs/xl storage type)
• cortex upgrade (Upgraded cortex to v1.14.1 #12935 )

[csengerszabo]

/assign @mohamed-rafraf

[embik]

/unassign @mohamed-rafraf
/assign

[embik]

Results so far:

• MLA upgrade worked well up to IAP upgrade fails with no such file or directory for test.sh symlink #13142.
• Applications just vanished from the user cluster. See All application installations installed from default app catalog were deleted on upgrade to v2.25.0-beta.2 #13143. I'll try to find more details retroactively, but I expect someone needs to do their own upgrade tests to observe this in detail. I honestly did not expect this to happen.
• AWS CCM crash loops, see AWS CCM crash loops after upgrade to v2.25.0-beta.2 and cloud spec cannot be upgraded #13144.

[embik]

Notes on the upgrade:

• The upgrade itself was pretty painless, I did not need to adjust any configuration.
• Upgrade guide should still feature all the MLA values changes. Since I had a minimal configuration it didn't affect me, but some users are probably customizing their MLA setup further.

[embik]

Upgrading from v2.24.4 to v2.25.0-rc.0:

• spec.cloud.aws.roleARN got migrated automatically to the fully qualified way. No CCM crashes observed.
• apps did not vanish from the user clusters and have their upgraded versions available from the version drop-down.
• logos are in the application catalog.
• no issue during MLA upgrade.

[embik]

Seems we are all good!