🐛 Ensure that webhook server is thread/start-safe #1155
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
approved
|
/retest |
|
Hi, it looks like our workaround to this problem was unsuccessful. Do you have any idea when this might make it into the next release of controller-runtime? Thanks, A |
This ensures that the webhook server is both threadsafe & "start-safe" -- i.e. you can register webhooks after starting the server. While this is generally not a common pattern, be allow runnables to be added to the manager after start, so it makes sense to do the same with hooks & the server.
DirectXMan12
force-pushed
the
bug/webhook-server-threadsafe
branch
from
ff5f036
to
5f1af13
Compare
|
Hmm... looks like there's a theoretical race in event broadcaster shutdown due to the event broadcaster calling race detector output |
|
/retest |
adrianludwin
added a commit
to adrianludwin/multi-tenancy
that referenced
this pull request
Sep 21, 2020
This essentially reverts kubernetes-retired#1087, which breaks HNC on new clusters that haven't previously had HNC installed. It fixes the nondeterministic crashing problem by patching in kubernetes-sigs/controller-runtime#1155, which has been applied to controller-runtime 0.6.3 in @adrianludwin's repo. This is a temporary hack and will be removed when controller-runtime releases its own fix - likely 0.6.4. Tested: with the reversion of kubernetes-retired#1087 (main.go), HNC can be installed on a fresh cluster again but fails to start up ~50% of the time. With the fix to controller-runtime, it passes on 20/20 startup attempts. Ran e2e tests and got the same result as without this change (four failures).
adrianludwin
added a commit
to adrianludwin/multi-tenancy
that referenced
this pull request
Sep 21, 2020
This essentially reverts kubernetes-retired#1087, which breaks HNC on new clusters that haven't previously had HNC installed. It fixes the nondeterministic crashing problem by patching in kubernetes-sigs/controller-runtime#1155, which has been applied to controller-runtime 0.6.3 in adrianludwin's repo. This is a temporary hack and will be removed when controller-runtime releases its own fix - likely 0.6.4. Tested: with the reversion of kubernetes-retired#1087 (main.go), HNC can be installed on a fresh cluster again but fails to start up ~50% of the time. With the fix to controller-runtime, it passes on 20/20 startup attempts. Ran e2e tests and got the same result as without this change (four failures).
DirectXMan12
requested review from
mengqiy and
vincepri
and removed request for
droot
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: DirectXMan12, mengqiy The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
vincepri
reviewed
Sep 22, 2020
Comment on lines
+149
to
+150
| // PrepWithoutInstalling does the setup parts of Install (populating host-port, | ||
| // setting up CAs, etc), without actually truing to do anything with webhook |
There was a problem hiding this comment.
Suggested change
| // PrepWithoutInstalling does the setup parts of Install (populating host-port, | |
| // setting up CAs, etc), without actually truing to do anything with webhook | |
| // PrepWithoutInstalling does the setup parts of Install (populating host-port, | |
| // setting up CAs, etc), without actually trying to do anything with webhook |
| // definitions. This is largely useful for internal testing of | ||
| // controller-runtime, where we need a random host-port & caData for webhook | ||
| // tests, but may be useful in similar scenarios. | ||
| func (o *WebhookInstallOptions) PrepWithoutInstalling() error { |
There was a problem hiding this comment.
Could we just call this Prepare?
Comment on lines
+356
to
+381
| server, wasNew := func() (*webhook.Server, bool) { | ||
| cm.mu.Lock() | ||
| defer cm.mu.Unlock() | ||
|
|
||
| if cm.webhookServer != nil { | ||
| return cm.webhookServer, false | ||
| } | ||
|
|
||
| cm.webhookServer = &webhook.Server{ | ||
| Port: cm.port, | ||
| Host: cm.host, | ||
| CertDir: cm.certDir, | ||
| } | ||
| if err := cm.Add(cm.webhookServer); err != nil { | ||
| panic("unable to add webhookServer to the controller manager") | ||
| return cm.webhookServer, true | ||
| }() | ||
|
|
||
| // only add the server if *we ourselves* just registered it. | ||
| // Add has its own lock, so just do this separately -- there shouldn't | ||
| // be a "race" in this lock gap because the condition is the population | ||
| // of cm.webhookServer, not anything to do with Add. | ||
| if wasNew { | ||
| if err := cm.Add(server); err != nil { | ||
| panic("unable to add webhook server to the controller manager") | ||
| } | ||
| } | ||
| return cm.webhookServer | ||
| return server |
There was a problem hiding this comment.
Unless I missed something, we should be able to simplify this method. Why do we need an inner function that's called right away?
There was a problem hiding this comment.
The IIFE sets the scope for the defer unlock call, which would otherwise be weird to unlock due to multiple return paths.
Comment on lines
+80
to
+81
| // mu protects access to the webhook map & setFields for Start, Register, etc | ||
| mu sync.Mutex |
There was a problem hiding this comment.
Nit: Define a mutex above the "protected" fields, so code becomes documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This ensures that the webhook server is both threadsafe & "start-safe"
-- i.e. you can register webhooks after starting the server. While this
is generally not a common pattern, be allow runnables to be added to the
manager after start, so it makes sense to do the same with hooks & the
server.
Fixes #1148
/kind bug