🌱 Add log level for kube components patch to ClusterClass #9493

killianmuldoon · 2023-09-25T15:55:58Z

Adds svariables and matching patches to set log levels for topology based clusters in e2e. The motivation for adding this is to help debug #8816.

The patch is enabled for that test in this PR, but this seems like a generally useful flag for helping debug e2e tests.

killianmuldoon

/area e2e-testing

@nawazkh @kubernetes-sigs/cluster-api-release-team

killianmuldoon · 2023-09-25T15:57:51Z

/test

k8s-ci-robot · 2023-09-25T15:57:53Z

@killianmuldoon: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test pull-cluster-api-build-main
/test pull-cluster-api-e2e-full-dualstack-and-ipv6-main
/test pull-cluster-api-e2e-full-main
/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-mink8s-main
/test pull-cluster-api-e2e-workload-upgrade-1-28-latest-main
/test pull-cluster-api-test-main
/test pull-cluster-api-test-mink8s-main
/test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-main
/test pull-cluster-api-e2e-scale-main-experimental

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-apidiff-main
pull-cluster-api-build-main
pull-cluster-api-e2e-main
pull-cluster-api-test-main
pull-cluster-api-verify-main

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

killianmuldoon · 2023-09-25T15:58:02Z

/test pull-cluster-api-e2e-full-dualstack-and-ipv6-main

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml

chrischdi

/lgtm

k8s-ci-robot · 2023-09-25T16:05:36Z

LGTM label has been added.

Git tree hash: bf926fb99a46ca66bc23bad02b9a1319d91a4691

killianmuldoon · 2023-09-25T16:07:19Z

I'll merge this once I've taken a look at the output logs in the dualstack tests where it's enabled.

fabriziopandini · 2023-09-25T16:26:50Z

/lgtm
we can eventually iterate later and make this more flexible

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml

killianmuldoon · 2023-09-25T17:19:26Z

/test pull-cluster-api-e2e-full-dualstack-and-ipv6-main

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml

killianmuldoon · 2023-09-25T17:32:12Z

/test pull-cluster-api-e2e-full-dualstack-and-ipv6-main

killianmuldoon · 2023-09-25T19:58:56Z

/test pull-cluster-api-e2e-full-dualstack-and-ipv6-main

killianmuldoon · 2023-09-26T10:49:10Z

Updated this so there's two variables - one for the kube control plane and another for kubelets. I think that's broken down enough for our needs - kubelet doesn't tend to log much in our tests as we don't schedule many pods. Log level 4 is way too high for the api server and the controller-manager though.

I looked at the kind tests where they set log level 4 and they end up with multiple files and don't have all of the logs from the test run AFAICT. Log level 2 seems like it will be useful for figuring out what's going wrong in the flake

killianmuldoon · 2023-09-26T11:41:33Z

Looks in working order now - the output with the higher log levels is here

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml

chrischdi · 2023-09-26T13:04:46Z

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml

+  - name: kubeControlPlaneLogLevel
+    schema:
+      openAPIV3Schema:
+        type: string
+        description: "Log level for kube-apiserver, kube-scheduler and kube-controller-manager"
+        default: "0"
+  - name: kubeletLogLevel
+    schema:
+      openAPIV3Schema:
+        type: string
+        description: "Log level for kubelets on control plane and worker nodes"
+        default: "0"


NIT!

Suggested change

- name: kubeControlPlaneLogLevel

schema:

openAPIV3Schema:

type: string

description: "Log level for kube-apiserver, kube-scheduler and kube-controller-manager"

default: "0"

- name: kubeletLogLevel

schema:

openAPIV3Schema:

type: string

description: "Log level for kubelets on control plane and worker nodes"

default: "0"

- name: kubeControlPlaneLogLevel

schema:

openAPIV3Schema:

type: string

description: "Log level for kube-apiserver, kube-scheduler and kube-controller-manager"

default: ""

- name: kubeletLogLevel

schema:

openAPIV3Schema:

type: string

description: "Log level for kubelets on control plane and worker nodes"

default: ""

I'd suggest to default to empty string and not patch if the value is empty string (enabledIf). This way we would always keep the default. (may require to split the patch with regards to the variables)

Should be done now

killianmuldoon · 2023-09-26T13:23:59Z

I'll also have to cherry-pick this back to v1.5 manually so we get coverage there for the flakes in the test. Can't auto-cherry-pick because of the MachinePool stuff.

killianmuldoon · 2023-09-26T13:34:01Z

/retest

killianmuldoon · 2023-09-26T13:37:01Z

I verified this locally - but I can add PR-Blocking to the test if you want to be sure it works. Just don't want to have to run the full dualstack suite again to see some log lines 😅

killianmuldoon · 2023-09-26T13:43:32Z

/hold

For CI run with blocking on the dualstack test.

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml

test/e2e/data/infrastructure-docker/v1.5/clusterclass-quick-start.yaml

killianmuldoon · 2023-09-26T14:09:43Z

Going to revert the PR-Blocking change: Logs from the run are here if you'd like to compare: https://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/9493/pull-cluster-api-e2e-main/1706665816436510720/artifacts/clusters/

Signed-off-by: killianmuldoon <kmuldoon@vmware.com>

killianmuldoon · 2023-09-26T14:20:00Z

/hold cancel

chrischdi

/lgtm

k8s-ci-robot · 2023-09-26T15:22:44Z

LGTM label has been added.

Git tree hash: e424ec3c5e608f0abdee5c3b3df1ee3fb21ed586

killianmuldoon · 2023-09-26T15:32:33Z

Thanks for the help with this one @chrischdi!

/approve

k8s-ci-robot · 2023-09-26T15:32:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: killianmuldoon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [killianmuldoon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 25, 2023

k8s-ci-robot requested review from jackfrancis and ykakarap September 25, 2023 15:56

killianmuldoon commented Sep 25, 2023

View reviewed changes

k8s-ci-robot added area/e2e-testing Issues or PRs related to e2e testing and removed do-not-merge/needs-area PR is missing an area label labels Sep 25, 2023

chrischdi reviewed Sep 25, 2023

View reviewed changes

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml Outdated Show resolved Hide resolved

chrischdi reviewed Sep 25, 2023

View reviewed changes

k8s-ci-robot assigned chrischdi Sep 25, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 25, 2023

k8s-ci-robot assigned fabriziopandini Sep 25, 2023

chrischdi reviewed Sep 25, 2023

View reviewed changes

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml Outdated Show resolved Hide resolved

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from 870074c to 8ecfb97 Compare September 25, 2023 17:17

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 25, 2023

k8s-ci-robot requested review from chrischdi and fabriziopandini September 25, 2023 17:17

chrischdi reviewed Sep 25, 2023

View reviewed changes

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml Outdated Show resolved Hide resolved

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from 8ecfb97 to 94eb7d9 Compare September 25, 2023 17:25

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from 94eb7d9 to 5f9b40a Compare September 25, 2023 19:58

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 26, 2023

killianmuldoon changed the title ~~🌱 Add debug logging patch to ClusterClass~~ 🌱 Add log level for kube components patch to ClusterClass Sep 26, 2023

chrischdi reviewed Sep 26, 2023

View reviewed changes

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from ccf8a93 to a43f40b Compare September 26, 2023 13:23

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from a43f40b to 82ceecc Compare September 26, 2023 13:27

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from 82ceecc to 5a64c88 Compare September 26, 2023 13:43

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 26, 2023

chrischdi reviewed Sep 26, 2023

View reviewed changes

test/e2e/data/infrastructure-docker/main/clusterclass-quick-start.yaml Outdated Show resolved Hide resolved

test/e2e/data/infrastructure-docker/v1.5/clusterclass-quick-start.yaml Outdated Show resolved Hide resolved

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from 5a64c88 to 5bdd5b5 Compare September 26, 2023 14:09

Add debug logging patch to ClusterClass

c6e270e

Signed-off-by: killianmuldoon <kmuldoon@vmware.com>

killianmuldoon force-pushed the pr-add-debug-logging-clusterclass branch from 5bdd5b5 to c6e270e Compare September 26, 2023 14:10

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 26, 2023

chrischdi reviewed Sep 26, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 26, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 26, 2023

killianmuldoon mentioned this pull request Sep 26, 2023

[release-1.5] 🌱 Add log level for kube components patch to ClusterClass #9501

Merged

k8s-ci-robot merged commit 8d0dc91 into kubernetes-sigs:main Sep 26, 2023
20 checks passed

k8s-ci-robot added this to the v1.6 milestone Sep 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Add log level for kube components patch to ClusterClass #9493

🌱 Add log level for kube components patch to ClusterClass #9493

killianmuldoon commented Sep 25, 2023 •

edited

killianmuldoon left a comment

killianmuldoon commented Sep 25, 2023

k8s-ci-robot commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

chrischdi left a comment

k8s-ci-robot commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

fabriziopandini commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

killianmuldoon commented Sep 26, 2023 •

edited

killianmuldoon commented Sep 26, 2023

chrischdi Sep 26, 2023

killianmuldoon Sep 26, 2023

killianmuldoon Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

chrischdi left a comment

k8s-ci-robot commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

k8s-ci-robot commented Sep 26, 2023

🌱 Add log level for kube components patch to ClusterClass #9493

🌱 Add log level for kube components patch to ClusterClass #9493

Conversation

killianmuldoon commented Sep 25, 2023 • edited

killianmuldoon left a comment

Choose a reason for hiding this comment

killianmuldoon commented Sep 25, 2023

k8s-ci-robot commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

chrischdi left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

fabriziopandini commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

killianmuldoon commented Sep 25, 2023

killianmuldoon commented Sep 26, 2023 • edited

killianmuldoon commented Sep 26, 2023

chrischdi Sep 26, 2023

Choose a reason for hiding this comment

killianmuldoon Sep 26, 2023

Choose a reason for hiding this comment

killianmuldoon Sep 26, 2023

Choose a reason for hiding this comment

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

chrischdi left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 26, 2023

killianmuldoon commented Sep 26, 2023

k8s-ci-robot commented Sep 26, 2023

killianmuldoon commented Sep 25, 2023 •

edited

killianmuldoon commented Sep 26, 2023 •

edited