Fix operator shutdown hanging when kvstore is enabled #24979
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
giorio94
added
kind/bug
|
/test |
|
ci-multicluster hit known failure #24990 |
|
/ci-multicluster |
|
test-1.27-net-next egressgw tests failed due to #24835 having been merged, as this one is not rebased. |
joamaki
approved these changes
Apr 24, 2023
thorn3r
approved these changes
Apr 27, 2023
86aa873 ("controller: add new ControllerParam CancelDoFuncOnUpdate") introduced the possibility of canceling the context of the controller's DoFunc when the controller is updated, which is then reset. There's an extremely rare chance that this context is accessed while being overwritten. Hence, let's grab a local copy while holding the mutex. Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
The Controller implementation currently supports the possibility of specifying a context among the parameters, which is propagated to the DoFunc. Though, once the context is canceled, the DoFunc is likely to never succeed again (unless it ignores it). Hence, let's pause the controller until it is either updated (refreshing the context) or properly stopped. Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
f487647 ("operator: Wait for informers to shut down when stopping") introduced a WaitGroup to wait for the shutdown of all goroutines started by `legacy.onStart`. When running in kvstore mode, the operator spawns one goroutine to synchronize global services to the kvstore, continuously reading from a channel. Since that channel is never closed, the goroutine does not terminate, preventing the correct shutdown of the operator. This commit fixes it propagating the context, and stopping the goroutine when it is canceled. Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
giorio94
force-pushed
the
mio/operator-shutdown-kvstore
branch
from
8751419
to
b81e664
Compare
|
/test Rerunning all tests after rebasing onto main, to fix the test-1.27-net-next failures. |
|
/ci-datapath A few matrix entries failed while creating the LVH VMs. Rerunning |
|
/test-runtime Hit know flake #22373 |
maintainer-s-little-helper
bot
added
the
ready-to-merge
joamaki
added
backport-pending/1.13
maintainer-s-little-helper
bot
moved this from Needs backport from main
to Backport pending to v1.13
in 1.13.3
julianwiedmann
added
backport-done/1.13
maintainer-s-little-helper
bot
moved this from Backport pending to v1.13
to Backport done to v1.13
in 1.13.3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reporting here the message of the last commit, which addresses the core issue, for convenience:
The first two commits instead, perform the following modifications to the controller implementation: