OCPBUGS-32348: Make logging configurable for on-prem components #4326

mkowalski · 2024-04-17T09:49:59Z

It came to our attention that the default logging configuration for on-prem components is very verbose, causing some customers to overload their log storage infrastructure.

To remediate the issue we have heavily limited what is logged, but without ability to raise a level to DEBUG. With this PR we are allowing to create a ConfigMap logging in the respective
openshift-$PLATFORM-infra namespace what will accept the following data to configure the levels accordingly

disable-keepalived-logs: true
enable-nodeip-debug: true

By default no such ConfigMap exists, so that keepalived logs are enabled and Node IP logs are disabled. Creation, as described above, allows to tweak the behaviour.

Fixes: OCPBUGS-32348
Fixes: OCPBUGS-32027

openshift-ci-robot · 2024-04-17T09:50:06Z

@mkowalski: This pull request references Jira Issue OCPBUGS-32348, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.0) matches configured target version for branch (4.16.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

It came to our attention that the default logging configuration for on-prem components is very verbose, causing some customers to overload their log storage infrastructure.

To remediate the issue we have heavily limited what is logged, but without ability to raise a level to DEBUG. With this PR we are allowing to create a ConfigMap logging in the respective
openshift-$PLATFORM-infra namespace what will accept the following data to configure the levels accordingly
disable-keepalived-logs: true
enable-nodeip-debug: true
By default no such ConfigMap exists, so that keepalived logs are enabled and Node IP logs are disabled. Creation, as described above, allows to tweak the behaviour.

Fixes: OCPBUGS-32348
Fixes: OCPBUGS-32027

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-04-17T09:50:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mkowalski
Once this PR has been reviewed and has the lgtm label, please assign jkyros for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mkowalski · 2024-04-17T12:26:31Z

/test e2e-metal-ipi

mkowalski · 2024-04-22T15:38:23Z

/cherry-pick release-4.15

openshift-cherrypick-robot · 2024-04-22T15:38:25Z

@mkowalski: once the present PR merges, I will cherry-pick it on top of release-4.15 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mkowalski · 2024-04-22T15:38:45Z

/test e2e-metal-ipi

zhaozhanqi · 2024-04-24T07:07:35Z

@mkowalski trying to pre-merge testing this PR with cluster-bot build openshift/machine-config-operator#4326

However the cluster cannot be deployed due to keepalived pod cannot be ready with error:

Apr 24 06:25:46 master-0 kubenswrapper[21487]: I0424 06:25:46.867046   21487 kubelet_getters.go:187] "Pod status updated" pod="openshift-kni-infra/keepalived-master-0" status="Pending"
Apr 24 06:25:48 master-0 kubenswrapper[21487]: E0424 06:25:48.783620   21487 kuberuntime_manager.go:1262] container &Container{Name:keepalived,Image:registry.build03.ci.openshift.org/ci-ln-ydcj9w2/stable@sha256:3d71a6c1be0e96c914c1f1f496593901763a530356268364cfa6c467e7aad656,Command:[/bin/bash -c #/bin/bash
Apr 24 06:25:48 master-0 kubenswrapper[21487]:   if pid=$(pgrep -o keepalived); then
Apr 24 06:25:48 master-0 kubenswrapper[21487]: reload_keepalived()
Apr 24 06:25:48 master-0 kubenswrapper[21487]:   if pid=$(pgrep -o keepalived); then
Apr 24 06:25:48 master-0 kubenswrapper[21487]:       /usr/sbin/keepalived -f /etc/keepalived/keepalived.conf --dont-fork --vrrp --log-detail --log-console &
Apr 24 06:25:48 master-0 kubenswrapper[21487]:         reload_keepalived
Apr 24 06:25:48 master-0 kubenswrapper[21487]: declare -r keepalived_sock="/var/run/keepalived/keepalived.sock"
Apr 24 06:25:48 master-0 kubenswrapper[21487]: export -f reload_keepalived
Apr 24 06:25:48 master-0 kubenswrapper[21487]: if [ -s "/etc/keepalived/keepalived.conf" ]; then
Apr 24 06:25:48 master-0 kubenswrapper[21487]:       /usr/sbin/keepalived -f /etc/keepalived/keepalived.conf --dont-fork --vrrp &
Apr 24 06:25:48 master-0 kubenswrapper[21487]:       /usr/sbin/keepalived -f /etc/keepalived/keepalived.conf --dont-fork --vrrp --log-detail --log-console &
Apr 24 06:25:48 master-0 kubenswrapper[21487]: rm -f "$keepalived_sock"
Apr 24 06:25:48 master-0 kubenswrapper[21487]: socat UNIX-LISTEN:${keepalived_sock},fork system:'bash -c msg_handler'
Apr 24 06:25:48 master-0 kubenswrapper[21487]: ],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:NSS_SDB_USE_CACHE,Value:no,ValueFrom:nil,},EnvVar{Name:DISABLE_KEEPALIVED_LOGS,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:logging,},Key:disable-keepalived-logs,Optional:*true,},SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{209715200 0} {<nil>}  BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:conf-dir,ReadOnly:false,MountPath:/etc/keepalived,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},VolumeMount{Name:run-dir,ReadOnly:false,MountPath:/var/run/keepalived,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:chroot-host,ReadOnly:false,MountPath:/host,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},VolumeMount{Name:kubeconfigvarlib,ReadOnly:false,MountPath:/var/lib/kubelet,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},VolumeMount{Name:nodeip-configuration,ReadOnly:false,MountPath:/run/nodeip-configuration,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[/bin/bash -c [ ! -s "/etc/keepalived/keepalived.conf" ] || (echo "" > /tmp/keepalived.data && kill -s SIGUSR1 "$(pgrep -o keepalived)" && for i in $(seq 5); do grep -q "VRRP Instance" /tmp/keepalived.data && exit 0 || sleep 1; done && exit 1)
Apr 24 06:25:48 master-0 kubenswrapper[21487]: ],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:20,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:FallbackToLogsOnError,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod keepalived-master-0_openshift-kni-infra(9219abad689cceeebedaa7bff3782dc3): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition
Apr 24 06:25:49 master-0 kubenswrapper[21487]: E0424 06:25:49.377289   21487 event.go:355] "Unable to write event (may retry after sleeping)" err="Post \"https://api-int.sriov.openshift-qe.sdn.com:6443/api/v1/namespaces/openshift-kni-infra/events\": dial tcp 192.168.111.5:6443: connect: no route to host" event="&Event{ObjectMeta:{keepalived-master-0.17c91d0c6d1a5620  openshift-kni-infra    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kni-infra,Name:keepalived-master-0,UID:9219abad689cceeebedaa7bff3782dc3,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{keepalived-monitor},},Reason:Pulled,Message:Container image \"registry.build03.ci.openshift.org/ci-ln-ydcj9w2/stable@sha256:812339cfaa9b57e93f0694b4a6276cc72b46c200598e88264810b51c90d7bf29\" already present on machine,Source:EventSource{Component:kubelet,Host:master-0,},FirstTimestamp:2024-04-24 04:31:12.398439968 +0000 UTC m=+25.721149527,LastTimestamp:2024-04-24 04:31:12.398439968 +0000 UTC m=+25.721149527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-0,}"
Apr 24 06:25:49 master-0 kubenswrapper[21487]: E0424 06:25:49.785891   21487 kuberuntime_manager.go:1262] container &Container{Name:keepalived-monitor,Image:registry.build03.ci.openshift.org/ci-ln-ydcj9w2/stable@sha256:812339cfaa9b57e93f0694b4a6276cc72b46c200598e88264810b51c90d7bf29,Command:[/bin/bash -c #/bin/bash
Apr 24 06:25:49 master-0 kubenswrapper[21487]: dynkeepalived /var/lib/kubelet/kubeconfig /config/keepalived.conf.tmpl /etc/keepalived/keepalived.conf --api-vips "${api_vips}" --ingress-vips "${ingress_vips}"
Apr 24 06:25:49 master-0 kubenswrapper[21487]: ],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:ENABLE_UNICAST,Value:yes,ValueFrom:nil,},EnvVar{Name:IS_BOOTSTRAP,Value:no,ValueFrom:nil,},EnvVar{Name:ENABLE_NODEIP_DEBUG,Value:,ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:logging,},Key:enable-nodeip-debug,Optional:*true,},SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{209715200 0} {<nil>}  BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:resource-dir,ReadOnly:false,MountPath:/config,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},VolumeMount{Name:kubeconfigvarlib,ReadOnly:false,MountPath:/var/lib/kubelet,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},VolumeMount{Name:conf-dir,ReadOnly:false,MountPath:/etc/keepalived,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},VolumeMount{Name:run-dir,ReadOnly:false,MountPath:/var/run/keepalived,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:chroot-host,ReadOnly:false,MountPath:/host,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},VolumeMount{Name:nodeip-configuration,ReadOnly:false,MountPath:/run/nodeip-configuration,SubPath:,MountPropagation:*HostToContainer,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:FallbackToLogsOnError,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod keepalived-master-0_openshift-kni-infra(9219abad689cceeebedaa7bff3782dc3): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition
Apr 24 06:25:49 master-0 kubenswrapper[21487]: E0424 06:25:49.785999   21487 pod_workers.go:1298] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"keepalived\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\", failed to \"StartContainer\" for \"keepalived-monitor\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"]" pod="openshift-kni-infra/keepalived-master-0" podUID="9219abad689cceeebedaa7bff3782dc3"
Apr 24 06:26:01 master-0 kubenswrapper[21487]: E0424 06:26:01.666377   21487 event.go:355] "Unable to write event (may retry after sleeping)" err="Post \"https://api-int.sriov.openshift-qe.sdn.com:6443/api/v1/namespaces/openshift-kni-infra/events\": dial tcp 192.168.111.5:6443: connect: no route to host" event="&Event{ObjectMeta:{keepalived-master-0.17c91d0c6d1a5620  openshift-kni-infra    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kni-infra,Name:keepalived-master-0,UID:9219abad689cceeebedaa7bff3782dc3,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{keepalived-monitor},},Reason:Pulled,Message:Container image \"registry.build03.ci.openshift.org/ci-ln-ydcj9w2/stable@sha256:812339cfaa9b57e93f0694b4a6276cc72b46c200598e88264810b51c90d7bf29\" already present on machine,Source:EventSource{Component:kubelet,Host:master-0,},FirstTimestamp:2024-04-24 04:31:12.398439968 +0000 UTC m=+25.721149527,LastTimestamp:2024-04-24 04:31:12.398439968 +0000 UTC m=+25.721149527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master-0,}"

mkowalski · 2024-04-24T11:28:58Z

Thanks for testing. Seems like indeed something is fishy with the new syntax

CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"

mkowalski · 2024-04-30T11:50:19Z

/hold

mkowalski · 2024-04-30T19:50:08Z

/test e2e-metal-ipi

mkowalski · 2024-04-30T19:50:13Z

/test e2e-openstack

mkowalski · 2024-04-30T20:25:23Z

/test e2e-metal-ipi

It came to our attention that the default logging configuration for on-prem components is very verbose, causing some customers to overload their log storage infrastructure. To remediate the issue we have heavily limited what is logged, but without ability to raise a level to DEBUG. With this PR we are allowing to create a ConfigMap `logging` in the respective `openshift-$PLATFORM-infra` namespace what will accept the following data to configure the levels accordingly ``` disable-keepalived-logs: true enable-nodeip-debug: true ``` By default no such ConfigMap exists, so that keepalived logs are enabled and Node IP logs are disabled. Creation, as described above, allows to tweak the behaviour. Fixes: OCPBUGS-32348 Fixes: OCPBUGS-32027

openshift-ci · 2024-04-30T21:19:28Z

@mkowalski: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`6c7e278`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Apr 17, 2024

openshift-ci bot requested review from zhaozhanqi, dkhater-redhat and engelmi April 17, 2024 09:50

openshift-ci-robot mentioned this pull request Apr 22, 2024

OCPBUGS-32348: Make log level configurable openshift/baremetal-runtimecfg#305

Merged

mkowalski force-pushed the OCPBUGS-32348 branch from 6c7e278 to 7bf981c Compare April 30, 2024 08:19

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 30, 2024

mkowalski force-pushed the OCPBUGS-32348 branch 2 times, most recently from 052e3b4 to 95f921e Compare April 30, 2024 20:24

mkowalski force-pushed the OCPBUGS-32348 branch from 95f921e to 7ca9d6b Compare April 30, 2024 21:18

mkowalski marked this pull request as draft April 30, 2024 21:21

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 30, 2024

openshift-ci-robot mentioned this pull request May 8, 2024

Revert "OCPBUGS-32348: Make log level configurable" openshift/baremetal-runtimecfg#310

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-32348: Make logging configurable for on-prem components #4326

OCPBUGS-32348: Make logging configurable for on-prem components #4326

mkowalski commented Apr 17, 2024

openshift-ci-robot commented Apr 17, 2024

openshift-ci bot commented Apr 17, 2024

mkowalski commented Apr 17, 2024

mkowalski commented Apr 22, 2024

openshift-cherrypick-robot commented Apr 22, 2024

mkowalski commented Apr 22, 2024

zhaozhanqi commented Apr 24, 2024

mkowalski commented Apr 24, 2024

mkowalski commented Apr 30, 2024

mkowalski commented Apr 30, 2024

mkowalski commented Apr 30, 2024

mkowalski commented Apr 30, 2024

openshift-ci bot commented Apr 30, 2024 •

edited

OCPBUGS-32348: Make logging configurable for on-prem components #4326

Are you sure you want to change the base?

OCPBUGS-32348: Make logging configurable for on-prem components #4326

Conversation

mkowalski commented Apr 17, 2024

openshift-ci-robot commented Apr 17, 2024

openshift-ci bot commented Apr 17, 2024

mkowalski commented Apr 17, 2024

mkowalski commented Apr 22, 2024

openshift-cherrypick-robot commented Apr 22, 2024

mkowalski commented Apr 22, 2024

zhaozhanqi commented Apr 24, 2024

mkowalski commented Apr 24, 2024

mkowalski commented Apr 30, 2024

mkowalski commented Apr 30, 2024

mkowalski commented Apr 30, 2024

mkowalski commented Apr 30, 2024

openshift-ci bot commented Apr 30, 2024 • edited

openshift-ci bot commented Apr 30, 2024 •

edited