Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix goroutine leak #709

Closed
piotrmiskiewicz opened this issue Apr 25, 2024 · 4 comments
Closed

Fix goroutine leak #709

piotrmiskiewicz opened this issue Apr 25, 2024 · 4 comments
Assignees

Comments

@piotrmiskiewicz
Copy link
Member

The number of goroutines is constantly growing. It looks like the AvS client (http with oauth) is created for every http call. It should be changed to use shared one. The initial AvS implementation was done this way, but there was a problem with token refreshing (see https://github.com/kyma-project/control-plane/pull/268/files).
AC:

  • no goroutines leak

In case of extracting AvShttp client to shared one - test the implementation for more than 24 h to check if it works fine (token refreshing etc.)

@MarekMichali
Copy link
Contributor

MarekMichali commented Apr 25, 2024

One way to test it on dev without promoting it to stage is to add the pr-deploy label to the promotion PR. This will allow you to push the tested KEB version to dev with promotion to stage blocked, and will also make the metrics available on the dashboard so we can see how it behaves. A good idea would be to do this on Friday and revert on Monday, so we get over 48 hours of testing without interfering with the normal KEB releases that happen during the week.

We should also create a custom KEB release from a fork for this testing scenario, by building and hosting the KEB chart there as we do upstream, without interfering with the upstream KEB release process. After successful testing, everything should be merged form the fork to upstream and released there.

@MarekMichali MarekMichali self-assigned this May 16, 2024
@MarekMichali
Copy link
Contributor

MarekMichali commented May 29, 2024

The leak is somewhere in AVS_Create_Internal_Eval_Step during both provisioning and deprovisioning. This was confirmed by disabling this step. After disabling it, the goroutines count stopped growing. This issue was also only seen on dev.

@MarekMichali
Copy link
Contributor

Creating AVS external evaluation is disabled on all environments, see #139

@MarekMichali
Copy link
Contributor

We decide to leave this issue for now, as it only occurs on dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants