OCPBUGS-31710: Unify machineconfig file format for kubeletconfig to u… #4350

QiWang19 · 2024-05-06T17:05:08Z

…se YAML encoding instead of JSON encoding
From doc: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/controller-revision-v1/#ControllerRevision about RawExtension, this field holds the unpacked data in json format. So keep using json encoding for kubeletconfig for unit/e2e testing.

- What I did
close: https://issues.redhat.com/browse/OCPBUGS-31710
- How to verify it
apply a kubelet config sets maxPods

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: max-pods-worker
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    maxPods: 107

sh-5.1# cat /etc/kubernetes/kubelet.conf
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
  x509:
    clientCAFile: /etc/kubernetes/kubelet-ca.crt
authorization:
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
cgroupRoot: /
clusterDNS:
- 172.30.0.10
clusterDomain: cluster.local
containerLogMaxSize: 50Mi
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
enableSystemLogQuery: true
evictionPressureTransitionPeriod: 0s
featureGates:
  AdminNetworkPolicy: true
  AlibabaPlatform: true
  AutomatedEtcdBackup: false
  AzureWorkloadIdentity: true
  BareMetalLoadBalancer: true
  BuildCSIVolumes: true
  CSIDriverSharedResource: false
  ChunkSizeMiB: false
  CloudDualStackNodeIPs: true
  ClusterAPIInstall: false
  ClusterAPIInstallAWS: true
  ClusterAPIInstallAzure: false
  ClusterAPIInstallGCP: false
  ClusterAPIInstallIBMCloud: false
  ClusterAPIInstallNutanix: true
  ClusterAPIInstallOpenStack: true
  ClusterAPIInstallPowerVS: false
  ClusterAPIInstallVSphere: true
  DNSNameResolver: false
  DisableKubeletCloudCredentialProviders: true
  DynamicResourceAllocation: false
  EtcdBackendQuota: false
  EventedPLEG: false
  Example: false
  ExternalCloudProvider: true
  ExternalCloudProviderAzure: true
  ExternalCloudProviderExternal: true
  ExternalCloudProviderGCP: true
  ExternalOIDC: false
  ExternalRouteCertificate: false
  GCPClusterHostedDNS: false
  GCPLabelsTags: false
  GatewayAPI: false
  HardwareSpeed: false
  ImagePolicy: false
  InsightsConfig: false
  InsightsConfigAPI: false
  InsightsOnDemandDataGather: false
  InstallAlternateInfrastructureAWS: false
  KMSv1: true
  MachineAPIOperatorDisableMachineHealthCheckController: false
  MachineAPIProviderOpenStack: false
  MachineConfigNodes: false
  ManagedBootImages: false
  MaxUnavailableStatefulSet: false
  MetricsCollectionProfiles: false
  MetricsServer: true
  MixedCPUsAllocation: false
  NetworkDiagnosticsConfig: true
  NetworkLiveMigration: true
  NewOLM: false
  NodeDisruptionPolicy: false
  NodeSwap: false
  OnClusterBuild: false
  OpenShiftPodSecurityAdmission: true
  PinnedImages: false
  PlatformOperators: false
  PrivateHostedZoneAWS: true
  RouteExternalCertificate: false
  ServiceAccountTokenNodeBinding: false
  ServiceAccountTokenNodeBindingValidation: false
  ServiceAccountTokenPodNodeInfo: false
  SignatureStores: false
  SigstoreImageVerification: false
  TranslateStreamCloseWebsocketRequests: false
  UpgradeStatus: false
  VSphereControlPlaneMachineSet: true
  VSphereDriverConfiguration: false
  VSphereMultiVCenters: false
  VSphereStaticIPs: true
  ValidatingAdmissionPolicy: false
  VolumeGroupSnapshot: false
fileCheckFrequency: 0s
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
kubeAPIBurst: 100
kubeAPIQPS: 50
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
maxPods: 107
memorySwap: {}
nodeStatusReportFrequency: 5m0s
nodeStatusUpdateFrequency: 10s
podPidsLimit: 4096
protectKernelDefaults: true
rotateCertificates: true
runtimeRequestTimeout: 0s
serializeImagePulls: false
serverTLSBootstrap: true
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
systemCgroups: /system.slice
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
tlsMinVersion: VersionTLS12
volumeStatsAggPeriod: 0s

- Description for the changelog

openshift-ci-robot · 2024-05-06T17:05:13Z

@QiWang19: This pull request references Jira Issue OCPBUGS-31710, which is invalid:

expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

…se YAML encoding instead of JSON encoding

- What I did
close: https://issues.redhat.com/browse/OCPBUGS-31710
- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

QiWang19 · 2024-05-06T18:12:37Z

/jira refresh

openshift-ci-robot · 2024-05-06T18:12:43Z

@QiWang19: This pull request references Jira Issue OCPBUGS-31710, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.0) matches configured target version for branch (4.16.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-05-06T21:58:50Z

@QiWang19: This pull request references Jira Issue OCPBUGS-31710, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.0) matches configured target version for branch (4.16.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

…se YAML encoding instead of JSON encoding
From doc: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/controller-revision-v1/#ControllerRevision about RawExtension, this field holds the unpacked data in json format. So keep using json encoding for kubeletconfig for unit/e2e testing.

- What I did
close: https://issues.redhat.com/browse/OCPBUGS-31710
- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

QiWang19 · 2024-05-08T14:59:43Z

/retest-required

…se YAML encoding instead of JSON encoding Signed-off-by: Qi Wang <qiwan@redhat.com>

openshift-ci · 2024-05-09T05:11:50Z

@QiWang19: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-ovn-upgrade-out-of-change	`8b0c738`	link	false	`/test e2e-azure-ovn-upgrade-out-of-change`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

QiWang19 · 2024-05-09T15:27:31Z

@yuqi-zhang could you review?

yuqi-zhang

Generally seems fine. We should probably reach out to the IBM team to see if this change affects them at all before we merge though.

One question inline

yuqi-zhang · 2024-05-15T19:57:10Z

pkg/controller/kubelet-config/kubelet_config_bootstrap_test.go

@@ -28,7 +28,7 @@ func TestRunKubeletBootstrap(t *testing.T) {
 				helpers.NewMachineConfigPool("custom", nil, customSelector, "v0"),
 			}

-			kcRaw, err := EncodeKubeletConfig(&kubeletconfigv1beta1.KubeletConfiguration{MaxPods: 100}, kubeletconfigv1beta1.SchemeGroupVersion)
+			kcRaw, err := EncodeKubeletConfig(&kubeletconfigv1beta1.KubeletConfiguration{MaxPods: 100}, kubeletconfigv1beta1.SchemeGroupVersion, runtime.ContentTypeJSON)


Just curious, why are some of the tests explicitly using JSON still?

This uses json format to set RawExtension filed in the test kubeletconfig object, otherwise the test failed with the controller returning a JSON unmarshal error.
I thought it because this doc about RawExtension
https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/controller-revision-v1/#ControllerRevision, this field holds the unpacked data in JSON format. So keep using json encoding for kubeletconfig only for unit/e2e testing.

ack, makes sense, thanks!

openshift-ci · 2024-05-15T20:00:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: QiWang19, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2024-05-21T20:27:05Z

@QiWang19: This pull request references Jira Issue OCPBUGS-31710, which is invalid:

expected the bug to target either version "4.17." or "openshift-4.17.", but it targets "4.16.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

…se YAML encoding instead of JSON encoding
From doc: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/controller-revision-v1/#ControllerRevision about RawExtension, this field holds the unpacked data in json format. So keep using json encoding for kubeletconfig for unit/e2e testing.

- What I did
close: https://issues.redhat.com/browse/OCPBUGS-31710
- How to verify it
apply a kubelet config sets maxPods

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
 name: max-pods-worker
spec:
 machineConfigPoolSelector:
   matchLabels:
     pools.operator.machineconfiguration.openshift.io/worker: ""
 kubeletConfig:
   maxPods: 107

sh-5.1# cat /etc/kubernetes/kubelet.conf
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
 anonymous:
   enabled: false
 webhook:
   cacheTTL: 0s
 x509:
   clientCAFile: /etc/kubernetes/kubelet-ca.crt
authorization:
 webhook:
   cacheAuthorizedTTL: 0s
   cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
cgroupRoot: /
clusterDNS:
- 172.30.0.10
clusterDomain: cluster.local
containerLogMaxSize: 50Mi
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
enableSystemLogQuery: true
evictionPressureTransitionPeriod: 0s
featureGates:
 AdminNetworkPolicy: true
 AlibabaPlatform: true
 AutomatedEtcdBackup: false
 AzureWorkloadIdentity: true
 BareMetalLoadBalancer: true
 BuildCSIVolumes: true
 CSIDriverSharedResource: false
 ChunkSizeMiB: false
 CloudDualStackNodeIPs: true
 ClusterAPIInstall: false
 ClusterAPIInstallAWS: true
 ClusterAPIInstallAzure: false
 ClusterAPIInstallGCP: false
 ClusterAPIInstallIBMCloud: false
 ClusterAPIInstallNutanix: true
 ClusterAPIInstallOpenStack: true
 ClusterAPIInstallPowerVS: false
 ClusterAPIInstallVSphere: true
 DNSNameResolver: false
 DisableKubeletCloudCredentialProviders: true
 DynamicResourceAllocation: false
 EtcdBackendQuota: false
 EventedPLEG: false
 Example: false
 ExternalCloudProvider: true
 ExternalCloudProviderAzure: true
 ExternalCloudProviderExternal: true
 ExternalCloudProviderGCP: true
 ExternalOIDC: false
 ExternalRouteCertificate: false
 GCPClusterHostedDNS: false
 GCPLabelsTags: false
 GatewayAPI: false
 HardwareSpeed: false
 ImagePolicy: false
 InsightsConfig: false
 InsightsConfigAPI: false
 InsightsOnDemandDataGather: false
 InstallAlternateInfrastructureAWS: false
 KMSv1: true
 MachineAPIOperatorDisableMachineHealthCheckController: false
 MachineAPIProviderOpenStack: false
 MachineConfigNodes: false
 ManagedBootImages: false
 MaxUnavailableStatefulSet: false
 MetricsCollectionProfiles: false
 MetricsServer: true
 MixedCPUsAllocation: false
 NetworkDiagnosticsConfig: true
 NetworkLiveMigration: true
 NewOLM: false
 NodeDisruptionPolicy: false
 NodeSwap: false
 OnClusterBuild: false
 OpenShiftPodSecurityAdmission: true
 PinnedImages: false
 PlatformOperators: false
 PrivateHostedZoneAWS: true
 RouteExternalCertificate: false
 ServiceAccountTokenNodeBinding: false
 ServiceAccountTokenNodeBindingValidation: false
 ServiceAccountTokenPodNodeInfo: false
 SignatureStores: false
 SigstoreImageVerification: false
 TranslateStreamCloseWebsocketRequests: false
 UpgradeStatus: false
 VSphereControlPlaneMachineSet: true
 VSphereDriverConfiguration: false
 VSphereMultiVCenters: false
 VSphereStaticIPs: true
 ValidatingAdmissionPolicy: false
 VolumeGroupSnapshot: false
fileCheckFrequency: 0s
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
kubeAPIBurst: 100
kubeAPIQPS: 50
logging:
 flushFrequency: 0
 options:
   json:
     infoBufferSize: "0"
 verbosity: 0
maxPods: 107
memorySwap: {}
nodeStatusReportFrequency: 5m0s
nodeStatusUpdateFrequency: 10s
podPidsLimit: 4096
protectKernelDefaults: true
rotateCertificates: true
runtimeRequestTimeout: 0s
serializeImagePulls: false
serverTLSBootstrap: true
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
systemCgroups: /system.slice
tlsCipherSuites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
tlsMinVersion: VersionTLS12
volumeStatsAggPeriod: 0s

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

lyman9966 · 2024-05-28T05:20:15Z

/label qe-approved

1)apply the following yaml:
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
name: custom-maxPods
spec:
machineConfigPoolSelector:
matchLabels:
#custom-kubelet-max: max-pods
pools.operator.machineconfiguration.openshift.io/worker: ""
kubeletConfig:
maxPods: 510

after the mcp finish updating, check /etc/kubernetes/kubelet.conf on host , it's yaml format:
% oc debug node/ci-ln-27nbj22-72292-r7fwz-worker-a-lbx7k
sh-5.1# chroot /host
sh-5.1# cat /etc/kubernetes/kubelet.conf
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
x509:
clientCAFile: /etc/kubernetes/kubelet-ca.crt
authorization:
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
cgroupRoot: /
clusterDNS:

172.30.0.10
clusterDomain: cluster.local
containerLogMaxSize: 50Mi
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
enableSystemLogQuery: true
evictionPressureTransitionPeriod: 0s
featureGates:
AdminNetworkPolicy: true
AlibabaPlatform: true
AutomatedEtcdBackup: false
AzureWorkloadIdentity: true
BareMetalLoadBalancer: true
BuildCSIVolumes: true
CSIDriverSharedResource: false
ChunkSizeMiB: false
CloudDualStackNodeIPs: true
ClusterAPIInstall: false
ClusterAPIInstallAWS: true
ClusterAPIInstallAzure: false
ClusterAPIInstallGCP: false
ClusterAPIInstallIBMCloud: false
ClusterAPIInstallNutanix: true
...
tlsCipherSuites:
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
tlsMinVersion: VersionTLS12
volumeStatsAggPeriod: 0s

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 6, 2024

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label May 6, 2024

openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label May 6, 2024

QiWang19 force-pushed the kcfg-yaml branch 2 times, most recently from 453d8c2 to 50923c9 Compare May 6, 2024 21:57

QiWang19 force-pushed the kcfg-yaml branch from 50923c9 to 59b19c7 Compare May 8, 2024 19:57

OCPBUGS-31710: Unify machineconfig file format for kubeletconfig to u…

8b0c738

…se YAML encoding instead of JSON encoding Signed-off-by: Qi Wang <qiwan@redhat.com>

QiWang19 force-pushed the kcfg-yaml branch from 59b19c7 to 8b0c738 Compare May 9, 2024 01:46

yuqi-zhang approved these changes May 15, 2024

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 15, 2024

openshift-ci-robot added jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. and removed jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels May 21, 2024

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label May 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-31710: Unify machineconfig file format for kubeletconfig to u… #4350

OCPBUGS-31710: Unify machineconfig file format for kubeletconfig to u… #4350

QiWang19 commented May 6, 2024 •

edited

openshift-ci-robot commented May 6, 2024

QiWang19 commented May 6, 2024

openshift-ci-robot commented May 6, 2024

openshift-ci-robot commented May 6, 2024

QiWang19 commented May 8, 2024

openshift-ci bot commented May 9, 2024

QiWang19 commented May 9, 2024

yuqi-zhang left a comment

yuqi-zhang May 15, 2024

QiWang19 May 15, 2024

yuqi-zhang May 15, 2024

openshift-ci bot commented May 15, 2024

openshift-ci-robot commented May 21, 2024

lyman9966 commented May 28, 2024

OCPBUGS-31710: Unify machineconfig file format for kubeletconfig to u… #4350

Are you sure you want to change the base?

OCPBUGS-31710: Unify machineconfig file format for kubeletconfig to u… #4350

Conversation

QiWang19 commented May 6, 2024 • edited

openshift-ci-robot commented May 6, 2024

QiWang19 commented May 6, 2024

openshift-ci-robot commented May 6, 2024

openshift-ci-robot commented May 6, 2024

QiWang19 commented May 8, 2024

openshift-ci bot commented May 9, 2024

QiWang19 commented May 9, 2024

yuqi-zhang left a comment

Choose a reason for hiding this comment

yuqi-zhang May 15, 2024

Choose a reason for hiding this comment

QiWang19 May 15, 2024

Choose a reason for hiding this comment

yuqi-zhang May 15, 2024

Choose a reason for hiding this comment

openshift-ci bot commented May 15, 2024

openshift-ci-robot commented May 21, 2024

lyman9966 commented May 28, 2024

QiWang19 commented May 6, 2024 •

edited