OCPBUGS-33489: cmd: ensure controllers are started after informers #4359

hexfusion · 2024-05-10T00:24:46Z

This PR addresses a potential race that can result in a panic when a cluster is first installed. The race is due to the need to start again the shared informer which included the tech preview APIs after we have verified the features are available, If this is not done the tech preview API types might not be available to the informer resulting in a panic.

The docs from SharedInformerFactory outline that this usage is ok.

//	// Creating informers can also be created after Start, but then
//	// Start must be called again:
//	anotherGenericInformer := factory.ForResource(resource)
//	factory.Start(ctx.Done())
type SharedInformerFactory interface {

But in some circumstances, the informer could introduce nil objects into the cache which end up resulting in a panic. This can happen in both MCC and MCD.

The fix proposed is to start the PinnedImageSet controllers AFTER we start the informers closing the window of failure.

manual tests:

6 clusters with this payload no reported issues.
5 e2e-gcp-op-techpreview cluster no error
2 clusterbot generated techpreview clusters with this payload

openshift-ci-robot · 2024-05-10T00:24:51Z

@hexfusion: This pull request references Jira Issue OCPBUGS-33489, which is invalid:

expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

testing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-05-10T00:25:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2024-05-10T00:26:20Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hexfusion
Once this PR has been reviewed and has the lgtm label, please assign jkyros for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hexfusion · 2024-05-10T00:30:09Z

/jira refresh

openshift-ci-robot · 2024-05-10T00:30:15Z

@hexfusion: This pull request references Jira Issue OCPBUGS-33489, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.0) matches configured target version for branch (4.16.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hexfusion · 2024-05-10T15:10:02Z

/hold

hexfusion · 2024-05-10T15:24:45Z

/test e2e-vsphere-ovn-techpreview

openshift-ci · 2024-05-10T15:25:31Z

@hexfusion: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test 4.12-upgrade-from-stable-4.11-images
/test cluster-bootimages
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test images
/test unit
/test verify

The following commands are available to trigger optional jobs:

/test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade
/test bootstrap-unit
/test e2e-aws-disruptive
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-fips-op
/test e2e-aws-ovn-upgrade-out-of-change
/test e2e-aws-ovn-workers-rhel8
/test e2e-aws-proxy
/test e2e-aws-serial
/test e2e-aws-single-node
/test e2e-aws-upgrade-single-node
/test e2e-aws-workers-rhel8
/test e2e-azure
/test e2e-azure-ovn-upgrade
/test e2e-azure-ovn-upgrade-out-of-change
/test e2e-azure-upgrade
/test e2e-gcp-op-techpreview
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-rt
/test e2e-gcp-rt-op
/test e2e-gcp-single-node
/test e2e-gcp-upgrade
/test e2e-metal-assisted
/test e2e-metal-ipi
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6
/test e2e-openstack
/test e2e-openstack-dualstack
/test e2e-openstack-externallb
/test e2e-openstack-parallel
/test e2e-ovirt
/test e2e-ovirt-upgrade
/test e2e-ovn-step-registry
/test e2e-vsphere
/test e2e-vsphere-upgrade
/test e2e-vsphere-upi
/test e2e-vsphere-upi-zones
/test e2e-vsphere-zones
/test okd-e2e-aws
/test okd-e2e-gcp-op
/test okd-e2e-upgrade
/test okd-e2e-vsphere
/test okd-images
/test okd-scos-images
/test security

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-machine-config-operator-master-bootstrap-unit
pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn
pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn-upgrade
pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn-upgrade-out-of-change
pull-ci-openshift-machine-config-operator-master-e2e-azure-ovn-upgrade-out-of-change
pull-ci-openshift-machine-config-operator-master-e2e-gcp-op
pull-ci-openshift-machine-config-operator-master-e2e-gcp-op-single-node
pull-ci-openshift-machine-config-operator-master-e2e-gcp-op-techpreview
pull-ci-openshift-machine-config-operator-master-e2e-hypershift
pull-ci-openshift-machine-config-operator-master-images
pull-ci-openshift-machine-config-operator-master-okd-images
pull-ci-openshift-machine-config-operator-master-security
pull-ci-openshift-machine-config-operator-master-unit
pull-ci-openshift-machine-config-operator-master-verify

In response to this:

/test e2e-vsphere-ovn-techpreview

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hexfusion · 2024-05-10T17:25:49Z

/test e2e-gcp-op-techpreview

openshift-ci-robot · 2024-05-10T17:35:36Z

@hexfusion: This pull request references Jira Issue OCPBUGS-33489, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.0) matches configured target version for branch (4.16.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

This PR addresses a potential race that can result in a panic when a cluster is first installed. The race is due to the need to start the shared informer which included the tech preview APIs. After we have verified the features are available,

The docs from SharedInformerFactory outline that this usage is ok.
//	// Creating informers can also be created after Start, but then
//	// Start must be called again:
//	anotherGenericInformer := factory.ForResource(resource)
//	factory.Start(ctx.Done())
type SharedInformerFactory interface {
But in some circumstances, the informer could introduce nil objects into the cache which end up resulting in a panic. This can happen in both MCC and MCD.

The fix proposed is to start the PinnedImageSet controllers after we start the informers closing the window of failure.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hexfusion · 2024-05-10T19:10:30Z

/test e2e-gcp-op-techpreview

hexfusion · 2024-05-10T23:25:48Z

/test e2e-gcp-op-techpreview

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

sergiordlr · 2024-05-14T09:53:03Z

/test e2e-gcp-op-techpreview
/test e2e-gcp-op-single-node

hexfusion · 2024-05-14T10:31:00Z

Working on another approach

sergiordlr · 2024-05-28T08:58:33Z

/test e2e-gcp-op-techpreview

openshift-ci · 2024-05-28T10:41:13Z

@hexfusion: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels May 10, 2024

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label May 10, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 10, 2024

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 10, 2024

openshift-ci bot requested a review from sergiordlr May 10, 2024 00:30

hexfusion marked this pull request as ready for review May 10, 2024 15:10

openshift-ci bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels May 10, 2024

openshift-ci bot requested review from cdoern and djoshy May 10, 2024 15:11

hexfusion force-pushed the start branch from 716d111 to a45d002 Compare May 10, 2024 20:44

cmd: ensure controllers are started after informers

20352eb

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

hexfusion force-pushed the start branch from a45d002 to 20352eb Compare May 10, 2024 23:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-33489: cmd: ensure controllers are started after informers #4359

OCPBUGS-33489: cmd: ensure controllers are started after informers #4359

hexfusion commented May 10, 2024 •

edited

openshift-ci-robot commented May 10, 2024

openshift-ci bot commented May 10, 2024

openshift-ci bot commented May 10, 2024

hexfusion commented May 10, 2024

openshift-ci-robot commented May 10, 2024

hexfusion commented May 10, 2024

hexfusion commented May 10, 2024

openshift-ci bot commented May 10, 2024

hexfusion commented May 10, 2024

openshift-ci-robot commented May 10, 2024

hexfusion commented May 10, 2024

hexfusion commented May 10, 2024

sergiordlr commented May 14, 2024

hexfusion commented May 14, 2024

sergiordlr commented May 28, 2024

openshift-ci bot commented May 28, 2024

OCPBUGS-33489: cmd: ensure controllers are started after informers #4359

Are you sure you want to change the base?

OCPBUGS-33489: cmd: ensure controllers are started after informers #4359

Conversation

hexfusion commented May 10, 2024 • edited

openshift-ci-robot commented May 10, 2024

openshift-ci bot commented May 10, 2024

openshift-ci bot commented May 10, 2024

hexfusion commented May 10, 2024

openshift-ci-robot commented May 10, 2024

hexfusion commented May 10, 2024

hexfusion commented May 10, 2024

openshift-ci bot commented May 10, 2024

hexfusion commented May 10, 2024

openshift-ci-robot commented May 10, 2024

hexfusion commented May 10, 2024

hexfusion commented May 10, 2024

sergiordlr commented May 14, 2024

hexfusion commented May 14, 2024

sergiordlr commented May 28, 2024

openshift-ci bot commented May 28, 2024

hexfusion commented May 10, 2024 •

edited