Disable P&F by default in the apiserver library

[dgrisonnet]

Priority and Fairness is currently enabled by default for any consumers of the apiserver library. This include the kube-apiserver as well as aggregated apiservers that are based on the generic apiserver library. For the latter it doesn't really make sense to have P&F enabled by default because the apiservers are for the majority not supposed to be accessed directly, but rather via the kube-apiserver. In case there is still a need to enable P&F at an aggregated apiserver level, they can continue to do it either via the --enable-priority-and-fairness CLI flag or by turning on the feature like we do for the kube-apiserver.

Disable priority and fairness by default for the consumers of the apiserver library. The feature will still be enabled on the kube-apiserver by default.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dgrisonnet
Once this PR has been reviewed and has the lgtm label, please assign jpbetz for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/controlplane/OWNERS
• staging/src/k8s.io/apiserver/pkg/server/options/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dgrisonnet]

/kind cleanup

[dgrisonnet]

cc @tkashem @MikeSpreitzer

I don't know how controversial this change will be, but aggregated apiservers consuming the apiserver/generic-apiserver libraries currently have to disable P&F themselves when in my opinion it would make more sense it to be disabled by default in the library.
The only drawback from keeping P&F on by default on an aggregated apiserver right now is that we have to mutate its ClusterRole to give permissions to list and watch flowschemas. It is not the worst I would admit, but still not ideal when it sounds like P&F isn't very beneficial inside aggregated apiservers.

[benluddy]

/cc

[cici37]

/triage accepted

[dgrisonnet]

e2e failure seems unrelated

/retest-required

[MikeSpreitzer]

I think that it makes sense for the libraries used for aggregated apiservers to have APF disabled by default.

The associated Issue here (#124533) also touches on another issue (#124655), I think the reported problem came from upgrading in the wrong order.

[liggitt]

I think that it makes sense for the libraries used for aggregated apiservers to have APF disabled by default.

I don't think this makes sense. These API servers should be configured by default to behave correctly whether they are contacted directly or via kube-apiserver.

If a specific server wants to differ from default admission / filter behavior, that's fine, but the existing defaults make sense to me.

[liggitt]

		// Disable PriorityAndFairness by default since it only makes sense to have
		// on by default for the kube-apiserver and not for the aggregated
		// apiservers that are reusing this library.

I don't agree with this comment

[dgrisonnet]

Who you mind expanding on that? From my understanding, I would expect that most users want to configure priority and fairness at the kube-apiserver level, and only very few use-cases would really benefit from configuring P&F at the aggregated apiserver level since they usually handle fewer resources and are very specific.

[liggitt]

it is equally important that system controllers get priority when talking to aggregated API servers. P&F operating at that server level as well helps ensure that.

[deads2k]

If a specific server wants to differ from default admission / filter behavior, that's fine, but the existing defaults make sense to me.

Expanding on the idea, the nicest configuration here is probably

1. explicitly on - server should fail to start if the API is inaccessible or unreadable
2. explicitly off - server shouldn't even attempt to run p&f
3. no opinion - server should attempt to run p&f, log an error if it is not successful and disable the capability

[dgrisonnet]

The no opinion configuration you mentioned would be nice to have since it would prevent aggregated apiserver maintainers from making the difficult decision between:

1. disabling the feature by default
2. enabling the feature by default at the cost of requiring users to have a Kubernetes 1.29 if they use the version of the aggregated apiserver with the 1.29+ libraries