Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

馃悰 Controller.Watch() should not store watches if already started #1163

Merged

Conversation

vincepri
Copy link
Member

The controller internal struct holds a list of watches
(as []watchDescription) when someone calls .Watch() to then start the
watches and informers once we're ready to call Start().

This behavior caused a memory leak in the case Watch was called after
a controller has already been started and if the source.Kind's cache was
either stopped or not available any longer. The leak was caused by the
watches internal slice holding on to all references to each watch ever
issued (and their respective caches).

Signed-off-by: Vince Prignano vincepri@vmware.com

/assign @alvaroaleman @DirectXMan12 @pwittrock

I'll backport the fix to release-0.5 and release-0.6 if we all agree that this is the fix we want. For consistency, I also nil'd the c.watches slice after Start() has been called and the controller has started.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 15, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 15, 2020
@vincepri
Copy link
Member Author

/cc @ncdc @fabriziopandini

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 15, 2020
The controller internal struct holds a list of watches
(as []watchDescription) when someone calls .Watch() to then start the
watches and informers once we're ready to call Start().

This behavior caused a memory leak in the case Watch was called after
a controller has already been started and if the source.Kind's cache was
either stopped or not available any longer. The leak was caused by the
watches internal slice holding on to all references to each watch ever
issued (and their respective caches).

Signed-off-by: Vince Prignano <vincepri@vmware.com>
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 15, 2020
@ncdc
Copy link
Contributor

ncdc commented Sep 15, 2020

LGTM

@alvaroaleman
Copy link
Member

@vincepri are you sure this properly solves the problem? Even when the Controller doesn't hold the Source anymore, the underlying informer will have an eventhandler attached if was ever in use, preventing it from being garbage collected - or am I missing something?

/lgtm
/hold

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Sep 15, 2020
@ncdc
Copy link
Contributor

ncdc commented Sep 15, 2020

@alvaroaleman we are dynamically creating a Cache and closing its stop channel when we're done with it, which GC's the informers.

@vincepri
Copy link
Member Author

What Andy said, if the stop channel is called, the informer properly stops, although the controller holds a reference to the KindSource which has a reference to the underlying cache, which will never be garbage collected

@alvaroaleman
Copy link
Member

okay, thanks for clarifying
/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 15, 2020
@k8s-ci-robot k8s-ci-robot merged commit 20af901 into kubernetes-sigs:master Sep 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants