Step 3 – generic controlplane: split server #120202

sttts · 2023-08-28T06:41:16Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This PR splits the kube-apiserver server struct Instance (pkg/controlplane/instance.go) into the kube-apiserver-specific and the generic one pkg/controlplane/apiserver.Server.

Part of #124530.

Which issue(s) this PR fixes:

Towards kubernetes/enhancements#4080

Special notes for your reviewer:

This is exclusively code move. No functional changes.
pkg/controlplane is a misnamer. It's actually highly kube-apiserver specific and will eventually be renamed to make the difference to pkg/controlplane/apiserver clear. But that's beyond this PR.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2023-08-28T06:41:25Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

linux-foundation-easycla · 2023-09-01T09:24:43Z

The committers listed above are authorized under a signed CLA.

✅ login: sttts / name: Dr. Stefan Schimanski (3b6d2a6)

k8s-triage-robot · 2024-04-27T09:52:46Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

sttts · 2024-04-27T16:22:03Z

pkg/controlplane/apiserver/apis.go

+		return fmt.Errorf("error in registering group versions: %w", err)
+	}
+	return nil
+}


For easier review:

// InstallAPIs will install the APIs for the restStorageProviders if they are enabled. -func (m *Instance) InstallAPIs(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter, restStorageProviders ...RESTStorageProvider) error { +func (s *Server) InstallAPIs(restStorageProviders ...RESTStorageProvider) error { nonLegacy := []*genericapiserver.APIGroupInfo{} // used later in the loop to filter the served resource by those that have expired. - resourceExpirationEvaluator, err := genericapiserver.NewResourceExpirationEvaluator(*m.GenericAPIServer.Version) + resourceExpirationEvaluator, err := genericapiserver.NewResourceExpirationEvaluator(*s.GenericAPIServer.Version) if err != nil { return err } for _, restStorageBuilder := range restStorageProviders { groupName := restStorageBuilder.GroupName() - apiGroupInfo, err := restStorageBuilder.NewRESTStorage(apiResourceConfigSource, restOptionsGetter) + apiGroupInfo, err := restStorageBuilder.NewRESTStorage(s.APIResourceConfigSource, s.RESTOptionsGetter) if err != nil { - return fmt.Errorf("problem initializing API group %q : %v", groupName, err) + return fmt.Errorf("problem initializing API group %q: %w", groupName, err) } if len(apiGroupInfo.VersionedResourcesStorageMap) == 0 { // If we have no storage for any resource configured, this API group is effectively disabled. @@ -35,14 +35,14 @@ if postHookProvider, ok := restStorageBuilder.(genericapiserver.PostStartHookProvider); ok { name, hook, err := postHookProvider.PostStartHook() if err != nil { - klog.Fatalf("Error building PostStartHook: %v", err) + return fmt.Errorf("error building PostStartHook: %w", err) } - m.GenericAPIServer.AddPostStartHookOrDie(name, hook) + s.GenericAPIServer.AddPostStartHookOrDie(name, hook) } if len(groupName) == 0 { // the legacy group for core APIs is special that it is installed into /api via this special install method. - if err := m.GenericAPIServer.InstallLegacyAPIGroup(genericapiserver.DefaultLegacyAPIPrefix, &apiGroupInfo); err != nil { + if err := s.GenericAPIServer.InstallLegacyAPIGroup(genericapiserver.DefaultLegacyAPIPrefix, &apiGroupInfo); err != nil { return fmt.Errorf("error in registering legacy API: %w", err) } } else { @@ -51,8 +51,8 @@ } } - if err := m.GenericAPIServer.InstallAPIGroups(nonLegacy...); err != nil { - return fmt.Errorf("error in registering group versions: %v", err) + if err := s.GenericAPIServer.InstallAPIGroups(nonLegacy...); err != nil { + return fmt.Errorf("error in registering group versions: %w", err) } return nil }

sttts · 2024-04-27T16:30:19Z

pkg/controlplane/apiserver/server.go

+		})
+	}
+
+	if utilfeature.DefaultFeatureGate.Enabled(features.UnknownVersionInteroperabilityProxy) {


For easier review:

+ _, publicServicePort, err := c.Generic.SecureServing.HostPort() + if err != nil { + return nil, fmt.Errorf("failed to get listener address: %w", err) + } + if utilfeature.DefaultFeatureGate.Enabled(features.UnknownVersionInteroperabilityProxy) { - peeraddress := getPeerAddress(c.ControlPlane.Extra.PeerAdvertiseAddress, c.ControlPlane.Generic.PublicAddress, publicServicePort) + peeraddress := getPeerAddress(c.Extra.PeerAdvertiseAddress, c.Generic.PublicAddress, publicServicePort) peerEndpointCtrl := peerreconcilers.New( - c.ControlPlane.Generic.APIServerID, + c.Generic.APIServerID, peeraddress, - c.ControlPlane.Extra.PeerEndpointLeaseReconciler, - c.Extra.EndpointReconcilerConfig.Interval, + c.Extra.PeerEndpointLeaseReconciler, + c.Extra.PeerEndpointReconcileInterval, client) if err != nil { return nil, fmt.Errorf("failed to create peer endpoint lease controller: %w", err) } - m.GenericAPIServer.AddPostStartHookOrDie("peer-endpoint-reconciler-controller", + s.GenericAPIServer.AddPostStartHookOrDie("peer-endpoint-reconciler-controller", func(hookContext genericapiserver.PostStartHookContext) error { peerEndpointCtrl.Start(hookContext.StopCh) return nil }) - m.GenericAPIServer.AddPreShutdownHookOrDie("peer-endpoint-reconciler-controller", + s.GenericAPIServer.AddPreShutdownHookOrDie("peer-endpoint-reconciler-controller", func() error { peerEndpointCtrl.Stop() return nil }) - // Add PostStartHooks for Unknown Version Proxy filter. - if c.ControlPlane.Extra.PeerProxy != nil { - m.GenericAPIServer.AddPostStartHookOrDie("unknown-version-proxy-filter", func(context genericapiserver.PostStartHookContext) error { - err := c.ControlPlane.Extra.PeerProxy.WaitForCacheSync(context.StopCh) + if c.Extra.PeerProxy != nil { + s.GenericAPIServer.AddPostStartHookOrDie("unknown-version-proxy-filter", func(context genericapiserver.PostStartHookContext) error { + err := c.Extra.PeerProxy.WaitForCacheSync(context.StopCh) return err }) } } - m.GenericAPIServer.AddPostStartHookOrDie("start-cluster-authentication-info-controller", func(hookContext genericapiserver.PostStartHookContext) error { - controller := clusterauthenticationtrust.NewClusterAuthenticationTrustController(m.ClusterAuthenticationInfo, client) + s.GenericAPIServer.AddPostStartHookOrDie("start-cluster-authentication-info-controller", func(hookContext genericapiserver.PostStartHookContext) error { + controller := clusterauthenticationtrust.NewClusterAuthenticationTrustController(s.ClusterAuthenticationInfo, client) // generate a context from stopCh. This is to avoid modifying files which are relying on apiserver // TODO: See if we can pass ctx to the current method ctx := wait.ContextForChannel(hookContext.StopCh) // prime values and start listeners - if m.ClusterAuthenticationInfo.ClientCA != nil { - m.ClusterAuthenticationInfo.ClientCA.AddListener(controller) - if controller, ok := m.ClusterAuthenticationInfo.ClientCA.(dynamiccertificates.ControllerRunner); ok { + if s.ClusterAuthenticationInfo.ClientCA != nil { + s.ClusterAuthenticationInfo.ClientCA.AddListener(controller) + if controller, ok := s.ClusterAuthenticationInfo.ClientCA.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value. if err := controller.RunOnce(ctx); err != nil { runtime.HandleError(err) @@ -46,9 +50,9 @@ go controller.Run(ctx, 1) } } - if m.ClusterAuthenticationInfo.RequestHeaderCA != nil { - m.ClusterAuthenticationInfo.RequestHeaderCA.AddListener(controller) - if controller, ok := m.ClusterAuthenticationInfo.RequestHeaderCA.(dynamiccertificates.ControllerRunner); ok { + if s.ClusterAuthenticationInfo.RequestHeaderCA != nil { + s.ClusterAuthenticationInfo.RequestHeaderCA.AddListener(controller) + if controller, ok := s.ClusterAuthenticationInfo.RequestHeaderCA.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value. if err := controller.RunOnce(ctx); err != nil { runtime.HandleError(err) @@ -62,15 +66,15 @@ }) if utilfeature.DefaultFeatureGate.Enabled(apiserverfeatures.APIServerIdentity) { - m.GenericAPIServer.AddPostStartHookOrDie("start-kube-apiserver-identity-lease-controller", func(hookContext genericapiserver.PostStartHookContext) error { + s.GenericAPIServer.AddPostStartHookOrDie("start-kube-apiserver-identity-lease-controller", func(hookContext genericapiserver.PostStartHookContext) error { // generate a context from stopCh. This is to avoid modifying files which are relying on apiserver // TODO: See if we can pass ctx to the current method ctx := wait.ContextForChannel(hookContext.StopCh) - leaseName := m.GenericAPIServer.APIServerID - holderIdentity := m.GenericAPIServer.APIServerID + "_" + string(uuid.NewUUID()) + leaseName := s.GenericAPIServer.APIServerID + holderIdentity := s.GenericAPIServer.APIServerID + "_" + string(uuid.NewUUID()) - peeraddress := getPeerAddress(c.ControlPlane.Extra.PeerAdvertiseAddress, c.ControlPlane.Generic.PublicAddress, publicServicePort) + peeraddress := getPeerAddress(c.Extra.PeerAdvertiseAddress, c.Generic.PublicAddress, publicServicePort) // must replace ':,[]' in [ip:port] to be able to store this as a valid label value controller := lease.NewController( clock.RealClock{}, @@ -82,28 +86,28 @@ leaseName, metav1.NamespaceSystem, // TODO: receive identity label value as a parameter when post start hook is moved to generic apiserver. - labelAPIServerHeartbeatFunc(KubeAPIServer, peeraddress)) + labelAPIServerHeartbeatFunc(name, peeraddress)) go controller.Run(ctx) return nil }) // TODO: move this into generic apiserver and make the lease identity value configurable - m.GenericAPIServer.AddPostStartHookOrDie("start-kube-apiserver-identity-lease-garbage-collector", func(hookContext genericapiserver.PostStartHookContext) error { + s.GenericAPIServer.AddPostStartHookOrDie("start-kube-apiserver-identity-lease-garbage-collector", func(hookContext genericapiserver.PostStartHookContext) error { go apiserverleasegc.NewAPIServerLeaseGC( client, IdentityLeaseGCPeriod, metav1.NamespaceSystem, - KubeAPIServerIdentityLeaseLabelSelector, + IdentityLeaseComponentLabelKey+"="+name, ).Run(hookContext.StopCh) return nil }) } - m.GenericAPIServer.AddPostStartHookOrDie("start-legacy-token-tracking-controller", func(hookContext genericapiserver.PostStartHookContext) error { + s.GenericAPIServer.AddPostStartHookOrDie("start-legacy-token-tracking-controller", func(hookContext genericapiserver.PostStartHookContext) error { go legacytokentracking.NewController(client).Run(hookContext.StopCh) return nil }) - return m, nil + return s, nil } func labelAPIServerHeartbeatFunc(identity string, peeraddress string) lease.ProcessLeaseFunc {

sttts · 2024-04-27T16:32:16Z

pkg/controlplane/apiserver/server.go

+	} else {
+		routes.NewOpenIDMetadataServer(md.ConfigJSON, md.PublicKeysetJSON).
+			Install(generic.Handler.GoRestfulContainer)
+	}


For easier review:

- if c.ControlPlane.Extra.EnableLogsSupport { - routes.Logs{}.Install(s.Handler.GoRestfulContainer) + if c.EnableLogsSupport { + routes.Logs{}.Install(generic.Handler.GoRestfulContainer) } // Metadata and keys are expected to only change across restarts at present, // so we just marshal immediately and serve the cached JSON bytes. md, err := serviceaccount.NewOpenIDMetadata( - c.ControlPlane.Extra.ServiceAccountIssuerURL, - c.ControlPlane.Extra.ServiceAccountJWKSURI, - c.ControlPlane.Generic.ExternalAddress, - c.ControlPlane.Extra.ServiceAccountPublicKeys, + c.ServiceAccountIssuerURL, + c.ServiceAccountJWKSURI, + c.Generic.ExternalAddress, + c.ServiceAccountPublicKeys, ) if err != nil { // If there was an error, skip installing the endpoints and log the @@ -18,7 +18,7 @@ msg := fmt.Sprintf("Could not construct pre-rendered responses for"+ " ServiceAccountIssuerDiscovery endpoints. Endpoints will not be"+ " enabled. Error: %v", err) - if c.ControlPlane.Extra.ServiceAccountIssuerURL != "" { + if c.ServiceAccountIssuerURL != "" { // The user likely expects this feature to be enabled if issuer URL is // set and the feature gate is enabled. In the future, if there is no // longer a feature gate and issuer URL is not set, the user may not @@ -30,5 +30,5 @@ } } else { routes.NewOpenIDMetadataServer(md.ConfigJSON, md.PublicKeysetJSON). - Install(s.Handler.GoRestfulContainer) + Install(generic.Handler.GoRestfulContainer) }

sttts · 2024-04-27T16:33:46Z

pkg/controlplane/apiserver/peer.go

+	} else {
+		return net.JoinHostPort(publicAddress.String(), strconv.Itoa(publicServicePort))
+	}
+}


For easier review:

// utility function to get the apiserver address that is used by peer apiservers to proxy // requests to this apiserver in case the peer is incapable of serving the request -func getPeerAddress(peerAdvertiseAddress peerreconcilers.PeerAdvertiseAddress, publicAddress net.IP, publicServicePort int) string { +func getPeerAddress(peerAdvertiseAddress reconcilers.PeerAdvertiseAddress, publicAddress net.IP, publicServicePort int) string { if peerAdvertiseAddress.PeerAdvertiseIP != "" && peerAdvertiseAddress.PeerAdvertisePort != "" { return net.JoinHostPort(peerAdvertiseAddress.PeerAdvertiseIP, peerAdvertiseAddress.PeerAdvertisePort) } else {

sttts · 2024-04-27T19:23:06Z

/retest

mjudeikis · 2024-04-29T04:17:54Z

looks straightforward move. waiting on #124576

sttts · 2024-04-29T07:13:09Z

/retest

Signed-off-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

sttts · 2024-04-29T08:16:45Z

#124576 merged. @mjudeikis if you can take another look

mjudeikis · 2024-04-29T08:28:02Z

test/integration/controlplane/apiserver_identity_test.go

@@ -84,7 +85,7 @@ func TestCreateLeaseOnStart(t *testing.T) {
 		leases, err := kubeclient.
 			CoordinationV1().
 			Leases(metav1.NamespaceSystem).
-			List(context.TODO(), metav1.ListOptions{LabelSelector: controlplane.KubeAPIServerIdentityLeaseLabelSelector})
+			List(context.TODO(), metav1.ListOptions{LabelSelector: controlplaneapiserver.IdentityLeaseComponentLabelKey + "=" + controlplane.KubeAPIServer})


Just for learning, why this changes?

KubeAPIServerIdentityLeaseLabelSelector was a strange constant, although the components were defined just next to it. Also we want other component names than kube-apiserver for the generic case.

mjudeikis · 2024-04-29T08:53:39Z

/lgtm
small changes for the win!

k8s-ci-robot · 2024-04-29T08:53:45Z

LGTM label has been added.

Git tree hash: 36bf9d664a9b54918075a070ff23446de540d779

luxas

Thanks @sttts!

Minor comments, mostly for the future
Indeed LGTM as well 👍

luxas · 2024-05-02T13:14:38Z

pkg/controlplane/apiserver/server.go

+	"k8s.io/kubernetes/pkg/serviceaccount"
+)
+
+var (


I know this was var before as well, but couldn't it be const?

luxas · 2024-05-02T13:16:13Z

pkg/controlplane/apiserver/server.go

+	VersionedInformers        clientgoinformers.SharedInformerFactory
+}
+
+// New returns a new instance of Master from the given config.


new instance of Server...

luxas · 2024-05-02T13:18:09Z

pkg/controlplane/apiserver/server.go

+// New returns a new instance of Master from the given config.
+// Certain config fields will be set to a default value if unset.
+// Certain config fields must be specified, including:
+// KubeletClientConfig


I guess this is not relevant anymore; either we remove this last comment, or specify the fields which are actually required.

luxas · 2024-05-02T13:21:37Z

pkg/controlplane/apiserver/server.go

+				IdentityLeaseRenewIntervalPeriod,
+				leaseName,
+				metav1.NamespaceSystem,
+				// TODO: receive identity label value as a parameter when post start hook is moved to generic apiserver.


I guess this is fixed now? 😄

luxas · 2024-05-02T13:23:23Z

pkg/controlplane/apiserver/server.go

+			go controller.Run(ctx)
+			return nil
+		})
+		// TODO: move this into generic apiserver and make the lease identity value configurable


do we want to propagate the name variable into the hook ID below, instead of having "kube-apiserver" in the name, or if the IDs do not need to be stable, I guess we can just change "start-kube-apiserver-identity-lease-garbage-collector" to "start-apiserver-identity-lease-garbage-collector"

luxas · 2024-05-02T13:28:15Z

pkg/controlplane/apiserver/server.go

+			}
+		}
+
+		go controller.Run(ctx, 1)


maybe it's just me, but it seems scary to have the same variable name here as for the "inner" controllers, that define "local" variables with the same name

I know the code was like this, but why not just (later) move this line up where controller was defined, and use non-overlapping names?

luxas · 2024-05-02T13:30:58Z

pkg/controlplane/apiserver/server.go

+		peerEndpointCtrl := peerreconcilers.New(
+			c.Generic.APIServerID,
+			peeraddress,
+			c.Extra.PeerEndpointLeaseReconciler,


I guess you could remove the Extra field reference now if you did not want that to be visible in the field path

luxas · 2024-05-02T13:36:07Z

pkg/controlplane/apiserver/server.go

+		})
+	}
+
+	s.GenericAPIServer.AddPostStartHookOrDie("start-legacy-token-tracking-controller", func(hookContext genericapiserver.PostStartHookContext) error {


why do we need this in a new generic API server, if it's "legacy"?

luxas · 2024-05-02T13:45:13Z

pkg/controlplane/apiserver/apis.go

+		return err
+	}
+
+	for _, restStorageBuilder := range restStorageProviders {


nit: (probably was before as well) unexpected to call it restStorageBuilder when it is in the restStorageProviders list, restStorageProvider would be more expected

luxas · 2024-05-02T13:48:38Z

pkg/controlplane/apiserver/server.go

+	client := kubernetes.NewForConfigOrDie(s.GenericAPIServer.LoopbackClientConfig)
+	if len(c.SystemNamespaces) > 0 {
+		s.GenericAPIServer.AddPostStartHookOrDie("start-system-namespaces-controller", func(hookContext genericapiserver.PostStartHookContext) error {
+			go systemnamespaces.NewController(c.SystemNamespaces, client, s.VersionedInformers.Core().V1().Namespaces()).Run(hookContext.StopCh)


probably this is the pattern we want to follow, to have "local" properties off commonly-used fields in the structs they are used, but just noting that we could just use c.VersionedInformers here, because it's not used anywhere else (for now at least)

similar case for ClusterAuthenticationInfo, it's only used in this setup function, not later, so technically it would not need to be in the struct

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Aug 28, 2023

k8s-ci-robot requested review from alculquicondor and dchen1107 August 28, 2023 06:41

carlory mentioned this pull request Aug 28, 2023

Add generic control plane staging repository kubernetes/enhancements#4080

Open

7 tasks

This was referenced Aug 31, 2023

epic: Kubernetes 1.28 kcp-dev/kcp#3003

Closed

Community Meeting August 31, 2023 kcp-dev/kcp#3002

Closed

sttts force-pushed the sttts-controlplane-config-split branch from 48eb016 to e4e3524 Compare September 1, 2023 09:24

sttts changed the base branch from master to release-1.28 September 1, 2023 09:40

sttts force-pushed the sttts-controlplane-config-split branch 3 times, most recently from ccb2c07 to 31a8223 Compare April 27, 2024 16:18

sttts commented Apr 27, 2024

View reviewed changes

sttts force-pushed the sttts-controlplane-config-split branch 2 times, most recently from 67d34d1 to 81dd49f Compare April 27, 2024 19:06

sttts force-pushed the sttts-controlplane-config-split branch 2 times, most recently from 2d50493 to 46f5ce6 Compare April 28, 2024 08:14

kubernetes deleted a comment from k8s-ci-robot Apr 28, 2024

sttts force-pushed the sttts-controlplane-config-split branch from 46f5ce6 to ff305aa Compare April 29, 2024 06:16

sttts mentioned this pull request Apr 29, 2024

Step 2 – controlplane/apiserver: move peer proxy code to allow generic aggregator construction #124576

Merged

pkg/controlplane: split apart generic server part of instance.go

3b6d2a6

Signed-off-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

sttts force-pushed the sttts-controlplane-config-split branch from ff305aa to 3b6d2a6 Compare April 29, 2024 08:15

mjudeikis reviewed Apr 29, 2024

View reviewed changes

k8s-ci-robot assigned mjudeikis Apr 29, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 29, 2024

k8s-ci-robot merged commit 3a68b84 into kubernetes:master Apr 29, 2024
15 checks passed

k8s-ci-robot added this to the v1.31 milestone Apr 29, 2024

luxas reviewed May 2, 2024

View reviewed changes

sttts changed the title ~~Step 2 – generic controlplane: split server~~ Step 3 – generic controlplane: split server May 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Step 3 – generic controlplane: split server #120202

Step 3 – generic controlplane: split server #120202

sttts commented Aug 28, 2023 •

edited

k8s-ci-robot commented Aug 28, 2023

linux-foundation-easycla bot commented Sep 1, 2023 •

edited

k8s-triage-robot commented Apr 27, 2024

sttts Apr 27, 2024

sttts Apr 27, 2024

sttts Apr 27, 2024

sttts Apr 27, 2024

sttts commented Apr 27, 2024

mjudeikis commented Apr 29, 2024

sttts commented Apr 29, 2024

sttts commented Apr 29, 2024

mjudeikis Apr 29, 2024

sttts Apr 29, 2024

mjudeikis commented Apr 29, 2024

k8s-ci-robot commented Apr 29, 2024

luxas left a comment

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

luxas May 2, 2024

Step 3 – generic controlplane: split server #120202

Step 3 – generic controlplane: split server #120202

Conversation

sttts commented Aug 28, 2023 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Aug 28, 2023

linux-foundation-easycla bot commented Sep 1, 2023 • edited

k8s-triage-robot commented Apr 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented Apr 27, 2024

mjudeikis commented Apr 29, 2024

sttts commented Apr 29, 2024

sttts commented Apr 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjudeikis commented Apr 29, 2024

k8s-ci-robot commented Apr 29, 2024

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented Aug 28, 2023 •

edited

linux-foundation-easycla bot commented Sep 1, 2023 •

edited