API fails to fetch EtcdRestores (and other resources?) in dedicated master/seed architecture

[embik]

What happened

When setting up a dedicated master/seed architecture with KKP 2.22.2 (or the latest release branch commits for release/v.22), the UI returns errors on the project overview page. The kubermatic-api pods involved log the following error:

{"level":"debug","time":"2023-05-12T12:21:42.319Z","caller":"kubermatic-api/main.go:599","msg":"response","body":"{\"error\":{\"code\":500,\"message\":\"no matches for kind \\\"EtcdRestore\\\" in version \\\"kubermatic.k8c.io/v1\\\"\"}}\n","status":500,"uri":"/api/v2/projects/f465kk5nv6/etcdrestores"}

@pkprzekwas and I have been debugging this, and we believe this is happening due to the shared REST mapper injected into all providers as the error is thrown by discovery involving the REST mapper in question:

dashboard/modules/api/cmd/kubermatic-api/main.go

Line 349 in c0c8288

etcdRestoreProjectProviderGetter := kubernetesprovider.EtcdRestoreProjectProviderFactory(mgr.GetRESTMapper(), seedKubeconfigGetter)

This shared REST mapper utilises the master cluster for information about resources, even when the clients generated with a reference to the REST mapper are for seeds. That decision made sense when it was made - all CRDs lived on both master and seed. Unfortunately, this assumption was invalidated by kubermatic/kubermatic#10903, which was shipped starting with KKP 2.22.0.

As a note: Usage of the master REST mapper was also a problem in kubermatic/kubermatic, e.g. as described in kubermatic/kubermatic#12188.

This is kind of part of kubermatic/kubermatic#12186.

Expected behavior

The API is capable of listing EtcdRestores and all other resources that only live on seed clusters.

How to reproduce

• Set up KKP 2.22 with separate/dedicated master and seed clusters
• Create a Project from the dashboard
• View that project; the error message should happen

Environment

• UI Version: v2.22.2
• API Version: v2.22.2
• Domain: n/a
• Others:

Current workaround

Apply all CRDs to master cluster.

Affected user persona

Users with large scale KKP setups as described in our documentation.

Business goal to be improved

Metric to be improved

[embik]

Note: I'm not sure we can fix this on short notice, I've started a thread to discuss if we want to fix this here on the dashboard side or on the kubermatic/kubermatic side.

[embik]

While we fixed this by bringing back all CRDs, I believe it still makes sense to progress on decoupling. As it stands, this issue is a prerequisite for bringing back the CRD split.

[kubermatic-bot]

Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubermatic-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubermatic-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubermatic-bot]

@kubermatic-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.