Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix a race condition in SharedInformer #59828

Merged
merged 2 commits into from Feb 14, 2018
Merged

Fix a race condition in SharedInformer #59828

merged 2 commits into from Feb 14, 2018

Conversation

krousey
Copy link
Contributor

@krousey krousey commented Feb 13, 2018

What this PR does / why we need it:

This fixes a race condition that can occur in the sharedIndexInformer

Which issue(s) this PR fixes:
Fixes #59822

Release note:

Fixed a race condition in k8s.io/client-go/tools/cache.SharedInformer that could violate the sequential delivery guarantee and cause panics on shutdown.

Kris added 2 commits February 13, 2018 11:38
Add a test case for the race in kubernetes#59822
be482ad
Add started state to the processor to protect against double starts
3c36d9e 
This prevents a race condition where the sharedIndexInformer was
causeing the processorListener's run and pop method to be started
twice. That violated the SharedInformer's interface guarantee of
sequential delivery and also caused panics on shutdown.
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 13, 2018
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 13, 2018
@krousey
Copy link
Contributor Author

krousey commented Feb 13, 2018

Without the fix, I was able to use golang.org/x/tools/cmd/stress to generate about 76 panics in 31000 runs with the following:

> go test -c
> stress cache.test -test.run=TestSharedInformerInitializationRace

After the fix in the second commit, I got 0 failures in 870,000 runs.

@jennybuckley
Copy link

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 13, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jennybuckley, krousey

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these OWNERS Files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-github-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 6590ea6 into kubernetes:master Feb 14, 2018
@smarterclayton
Copy link
Contributor

How far does this need to be backported? @kubernetes/sig-api-machinery-bugs

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. kind/bug Categorizes issue or PR as related to a bug. labels Feb 14, 2018
@krousey
Copy link
Contributor Author

krousey commented Feb 14, 2018

@smarterclayton I came upon this while testing a controller using the 1.8 release libraries. I would like at least that far back to make my life easier.

@wojtek-t
Copy link
Member

@krousey - great finding! LGTM

@krousey krousey added this to the v1.8 milestone Feb 14, 2018
@jpbetz jpbetz added cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. and removed cherrypick-candidate labels Feb 14, 2018
@jpbetz jpbetz mentioned this pull request Feb 14, 2018
Merged
k8s-github-robot pushed a commit that referenced this pull request Feb 15, 2018
Merge pull request #59891 from jpbetz/automated-cherry-pick-of-#59828…
fa557ee 
…-origin-release-1.8

Automatic merge from submit-queue.

Automated cherry pick of #59828: Add a test case for the race in #59822

Cherry pick of #59828 on release-1.8.

#59828: Add a test case for the race in #59822
@krousey krousey modified the milestones: v1.8, v1.9 Feb 15, 2018
@krousey krousey removed the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Feb 15, 2018
@mbohlool
Copy link
Contributor

/cc @roycaihw @wenjiaswe

@k8s-ci-robot
Copy link
Contributor

@mbohlool: GitHub didn't allow me to request PR reviews from the following users: wenjiaswe.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @roycaihw @wenjiaswe

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@roycaihw
Copy link
Member

/lgtm

@deads2k deads2k mentioned this pull request Feb 19, 2018
Closed
@krousey
Copy link
Contributor Author

krousey commented Feb 20, 2018

@mbohlool can we approve this for the 1.9 branch yet?

@k8s-github-robot
Copy link

Commit found in the "release-1.9" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

@rmmh
Copy link
Contributor

rmmh commented Mar 27, 2018

re-adding cherrypick-candidate for debugging

@maisem maisem mentioned this pull request Mar 27, 2018
Merged
k8s-github-robot pushed a commit that referenced this pull request Mar 27, 2018
Merge pull request #61751 from maisem/automated-cherry-pick-of-#59828…
c996d04 
…-upstream-release-1.9

Automatic merge from submit-queue.

Automated cherry pick of #59828 upstream release 1.9

Cherry pick of #59828 on release-1.9.

#59828 : Fix a race condition in SharedInformer
@tossmilestone tossmilestone mentioned this pull request Jul 2, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@wojtek-t wojtek-t Awaiting requested review from wojtek-t

@roycaihw roycaihw Awaiting requested review from roycaihw

Assignees

@jennybuckley jennybuckley

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.9
Development

Successfully merging this pull request may close these issues.

Shared informer has a race condition
10 participants
@krousey @jennybuckley @k8s-ci-robot @k8s-github-robot @smarterclayton @wojtek-t @mbohlool @roycaihw @rmmh @jpbetz