Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inctimer: fix test flake where timer does not fire within time. #25219

Merged
merged 1 commit into from
May 2, 2023

Conversation

tommyp1ckles
Copy link
Contributor

@tommyp1ckles tommyp1ckles commented May 1, 2023

Running the test in a cpu constrained environment, such as:

docker run -v $(pwd):$(pwd) -w $(pwd) --cpus=0.1 -it golang:bullseye ./inctimer.test -test.v

I can fairly consistency reproduce a flake where the inctimer.After does not fire in time.
If I allow it to wait for an additional couple of ms, this seems to be sufficient to prevent failure.

It appears that goroutine scheduling latency can be significantly delayed in cpu restricted environments.
This seems unavoidable, so to fix the flake I'll allow the test to wait another 2ms to see if the inctimer eventually fires.

This will also log an error for delayed test fires, so if there is any other issues we can more easily debug them in the future.

Signed-off-by: Tom Hadlaw tom.hadlaw@isovalent.com

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 1, 2023
@tommyp1ckles
Copy link
Contributor Author

/test

Running the test in a cpu constrained environment, such as:

```
docker run -v $(pwd):$(pwd) -w $(pwd) --cpus=0.1 -it golang:bullseye ./inctimer.test -test.v
```

I can fairly consistency reproduce a flake where the inctimer.After does not fire in time.
If I allow it to wait for an additional couple of ms, this seems to be sufficient to prevent failure.

It appears that goroutine scheduling latency can be significantly delayed in cpu restricted environments.
This seems unavoidable, so to fix the flake I'll allow the test to wait another 2ms to see if the inctimer eventually fires.

This will also log an error for delayed test fires, so if there is any other issues we can more easily debug them in the future.

Fixed: cilium#25202

Signed-off-by: Tom Hadlaw <tom.hadlaw@isovalent.com>
@tommyp1ckles tommyp1ckles marked this pull request as ready for review May 1, 2023 20:37
@tommyp1ckles tommyp1ckles requested a review from a team as a code owner May 1, 2023 20:37
@tommyp1ckles tommyp1ckles added the kind/bug/CI This is a bug in the testing code. label May 1, 2023
@tommyp1ckles
Copy link
Contributor Author

/test

@tommyp1ckles tommyp1ckles added the release-note/ci This PR makes changes to the CI. label May 1, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 1, 2023
@tommyp1ckles tommyp1ckles added needs-backport/1.11 backport/1.12 This PR represents a backport for Cilium 1.12.x of a PR that was merged to main. needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch labels May 1, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot added this to Needs backport from main in 1.13.3 May 1, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot added this to Needs backport from main in 1.11.17 May 1, 2023
@pchaigno pchaigno added needs-backport/1.12 and removed backport/1.12 This PR represents a backport for Cilium 1.12.x of a PR that was merged to main. labels May 1, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot added this to Needs backport from main in 1.12.10 May 1, 2023
@tommyp1ckles tommyp1ckles added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label May 2, 2023
@joestringer joestringer merged commit e695e48 into cilium:main May 2, 2023
56 checks passed
@joamaki joamaki mentioned this pull request May 3, 2023
8 tasks
@joamaki joamaki mentioned this pull request May 3, 2023
8 tasks
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Needs backport from main to Backport pending to v1.12 in 1.12.10 May 3, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Needs backport from main to Backport pending to v1.11 in 1.11.17 May 3, 2023
@YutaroHayakawa YutaroHayakawa mentioned this pull request May 10, 2023
14 tasks
@YutaroHayakawa YutaroHayakawa added backport-pending/1.13 The backport for Cilium 1.13.x for this PR is in progress. and removed needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch labels May 10, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Needs backport from main to Backport pending to v1.13 in 1.13.3 May 10, 2023
@jrajahalme jrajahalme added backport-done/1.11 The backport for Cilium 1.11.x for this PR is done. and removed backport-pending/1.11 labels May 12, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.11 to Backport done to v1.11 in 1.11.17 May 12, 2023
@julianwiedmann julianwiedmann added backport-done/1.12 The backport for Cilium 1.12.x for this PR is done. backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. and removed backport-pending/1.12 backport-pending/1.13 The backport for Cilium 1.13.x for this PR is in progress. labels Jul 10, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.13 to Backport done to v1.13 in 1.13.3 Jul 10, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.12 to Backport done to v1.12 in 1.12.10 Jul 10, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.12 to Backport done to v1.12 in 1.12.10 Jul 10, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.12 to Backport done to v1.12 in 1.12.10 Jul 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-done/1.11 The backport for Cilium 1.11.x for this PR is done. backport-done/1.12 The backport for Cilium 1.12.x for this PR is done. backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. kind/bug/CI This is a bug in the testing code. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/ci This PR makes changes to the CI.
Projects
No open projects
1.11.17
Backport done to v1.11
1.12.10
Backport done to v1.12
1.13.3
Backport done to v1.13
Development

Successfully merging this pull request may close these issues.

None yet

7 participants