OCPBUGS-32242: The MCO makes a lot of API requests for ClusterOperator objects

[djoshy]

I added a lister and cached the CO object wherever I could, and only did a get call when there is an update conflict error. Initial results look promising. I think we should be safe to merge this if the test suite is passing, but it can wait till after FF. It does not introduce any new behavior, so we could probably skip pre-merge QE.

For a recentish test run(LARGE FILE WARNING) on #4327 :

{
                    "GroupVersionResource": {
                        "Group": "config.openshift.io",
                        "Version": "v1",
                        "Resource": "clusteroperators"
                    },
                    "RequestCounts": {
                        "SerializedRequestCounts": {
                            "RequestStartedCount": 0,
                            "RequestFinishedCount": 8034,
                            "ClientFailedRequestCount": 0,
                            "ServerFailedRequestCount": 0,
                            "PerHTTPStatusRequestCount": [
                                {
                                    "HTTPStatus": 200,
                                    "Count": 8042
                                }
                            ]
                        },
                        "PerVerbRequestCount": [
                            {
                                "Verb": "get",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 5933,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 5933
                                        }
                                    ]
                                }
                            },
                            {
                                "Verb": "update",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 2094,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 2094
                                        }
                                    ]
                                }
                            },
                            {
                                "Verb": "watch",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 7,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 15
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                },

On a test run on this PR, the above scrape only shows:

{
                    "GroupVersionResource": {
                        "Group": "config.openshift.io",
                        "Version": "v1",
                        "Resource": "clusteroperators"
                    },
                    "RequestCounts": {
                        "SerializedRequestCounts": {
                            "RequestStartedCount": 0,
                            "RequestFinishedCount": 7,
                            "ClientFailedRequestCount": 0,
                            "ServerFailedRequestCount": 0,
                            "PerHTTPStatusRequestCount": [
                                {
                                    "HTTPStatus": 200,
                                    "Count": 15
                                }
                            ]
                        },
                        "PerVerbRequestCount": [
                            {
                                "Verb": "watch",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 7,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 15
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                },

This is likely because a large amount of the calls have been optimized out; and some amount of calls when the cluster is born have been scrolled off in the audit logs and was not caught by the scraper tool. We should be failing most of the tests if we are not doing any update/get calls :P

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-32242, which is invalid:

• expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[djoshy]

/test unit
/test verify
/test e2e-gcp-op

[djoshy]

/test unit
/test verify
/test e2e-gcp-op

[djoshy]

/test unit
/test verify
/test e2e-gcp-op

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-32242. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-32242, which is invalid:

• expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/test unit
/test verify
/test e2e-gcp-op

[djoshy]

/test unit
/test verify
/test e2e-gcp-op

[djoshy]

/test unit
/test verify
/test e2e-gcp-op

[djoshy]

/test e2e-aws-ovn

[djoshy]

/test unit
/test verify
/test e2e-gcp-op
/test e2e-aws-ovn

[djoshy]

/test e2e-aws-ovn

[djoshy]

/test unit
/test verify
/test e2e-gcp-op
/test e2e-aws-ovn

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [djoshy,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

[openshift-ci-robot]

@openshift-bot: This pull request references Jira Issue OCPBUGS-32242, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

[openshift-ci-robot]

@openshift-bot: This pull request references Jira Issue OCPBUGS-32242, which is invalid:

• expected the bug to target either version "4.17." or "openshift-4.17.", but it targets "4.16.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/jira refresh

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-32242, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.17.0) matches configured target version for branch (4.17.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/test all

[sergiordlr]

Critical e2e test cases executed in: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-aws-ipi-f9-longduration-mco-critical/1792602290306158592 (failed, could not install)

[sergiordlr]

Critical e2e test cases executed in: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-aws-ipi-f9-longduration-mco-critical/1792826455558721536 (installation failed)

[sergiordlr]

Critical test cases passed: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-nightly-aws-ipi-f9-longduration-mco-critical/1792945702985273344

[sergiordlr]

Verfication steps

1. Critical test cases have passed
2. In the current CI job "e2e-aws-ovn" execution in we can find in the artifacts the audit-log-summary file https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_machine-config-operator/4321/pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn/1792546860762664960/artifacts/e2e-aws-ovn/openshift-e2e-test/artifacts/junit/audit-log-summary__20240520-141441.json

We can observe that for the clusteroperators

                {
                    "GroupVersionResource": {
                        "Group": "config.openshift.io",
                        "Version": "v1",
                        "Resource": "clusteroperators"
                    },
                    "RequestCounts": {
                        "SerializedRequestCounts": {
                            "RequestStartedCount": 0,
                            "RequestFinishedCount": 4,
                            "ClientFailedRequestCount": 0,
                            "ServerFailedRequestCount": 0,
                            "PerHTTPStatusRequestCount": [
                                {
                                    "HTTPStatus": 200,
                                    "Count": 9
                                }
                            ]
                        },
                        "PerVerbRequestCount": [
                            {
                                "Verb": "watch",
                                "RequestCounts": {
                                    "RequestStartedCount": 0,
                                    "RequestFinishedCount": 4,
                                    "ClientFailedRequestCount": 0,
                                    "ServerFailedRequestCount": 0,
                                    "PerHTTPStatusRequestCount": [
                                        {
                                            "HTTPStatus": 200,
                                            "Count": 9
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                },

Those numbers are lower than the ones found in previous executions and reported at the beginning of this PR.

/label qe-approved

[djoshy]

/unhold

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD e50be3e and 2 for PR HEAD 461a468 in total

[djoshy]

/test all

[djoshy]

/test e2e-gcp-op-single-node

[openshift-ci-robot]

@djoshy: Jira Issue OCPBUGS-32242: All pull requests linked via external trackers have merged:

• openshift/machine-config-operator#4321

Jira Issue OCPBUGS-32242 has been moved to the MODIFIED state.

In response to this:

I added a lister and cached the CO object wherever I could, and only did a get call when there is an update conflict error. Initial results look promising. I think we should be safe to merge this if the test suite is passing, but it can wait till after FF. It does not introduce any new behavior, so we could probably skip pre-merge QE.

For a recentish test run(LARGE FILE WARNING) on #4327 :

{
                   "GroupVersionResource": {
                       "Group": "config.openshift.io",
                       "Version": "v1",
                       "Resource": "clusteroperators"
                   },
                   "RequestCounts": {
                       "SerializedRequestCounts": {
                           "RequestStartedCount": 0,
                           "RequestFinishedCount": 8034,
                           "ClientFailedRequestCount": 0,
                           "ServerFailedRequestCount": 0,
                           "PerHTTPStatusRequestCount": [
                               {
                                   "HTTPStatus": 200,
                                   "Count": 8042
                               }
                           ]
                       },
                       "PerVerbRequestCount": [
                           {
                               "Verb": "get",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 5933,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 5933
                                       }
                                   ]
                               }
                           },
                           {
                               "Verb": "update",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 2094,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 2094
                                       }
                                   ]
                               }
                           },
                           {
                               "Verb": "watch",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 7,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 15
                                       }
                                   ]
                               }
                           }
                       ]
                   }
               },

On a test run on this PR, the above scrape only shows:

{
                   "GroupVersionResource": {
                       "Group": "config.openshift.io",
                       "Version": "v1",
                       "Resource": "clusteroperators"
                   },
                   "RequestCounts": {
                       "SerializedRequestCounts": {
                           "RequestStartedCount": 0,
                           "RequestFinishedCount": 7,
                           "ClientFailedRequestCount": 0,
                           "ServerFailedRequestCount": 0,
                           "PerHTTPStatusRequestCount": [
                               {
                                   "HTTPStatus": 200,
                                   "Count": 15
                               }
                           ]
                       },
                       "PerVerbRequestCount": [
                           {
                               "Verb": "watch",
                               "RequestCounts": {
                                   "RequestStartedCount": 0,
                                   "RequestFinishedCount": 7,
                                   "ClientFailedRequestCount": 0,
                                   "ServerFailedRequestCount": 0,
                                   "PerHTTPStatusRequestCount": [
                                       {
                                           "HTTPStatus": 200,
                                           "Count": 15
                                       }
                                   ]
                               }
                           }
                       ]
                   }
               },

This is likely because a large amount of the calls have been optimized out; and some amount of calls when the cluster is born have been scrolled off in the audit logs and was not caught by the scraper tool. We should be failing most of the tests if we are not doing any update/get calls :P

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@djoshy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	6e194c2	link	false	/test okd-scos-e2e-aws-ovn
ci/prow/okd-images	461a468	link	false	/test okd-images

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-machine-config-operator-container-v4.17.0-202405230209.p0.ge7da460.assembly.stream.el9 for distgit ose-machine-config-operator.
All builds following this will include this PR.