jenkinsfiles: Fix order of ginkgo tests #25002
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pchaigno
force-pushed
the
ginkgo-fix-test-order
branch
from
53c7e1c
to
1624dd1
Compare
The ginkgo tests are currently executed in a different order for each run in our CI. Unfortunately, the order in which we run the tests often matters and can impact the test results. That's because there are a lot of side effects (leftover Linux state, leftover Cilium conntrack entries, leftover Kubernetes state, etc.) that are hard to control for. This undeterminism facilities the introduction of flakes: a first pull request introduces a flake, but it doesn't show up until a later pull request runs the tests in a particular order. The random order also makes flake a bit order to reproduce locally, though that can easily be worked around by passing the proper seed to ginkgo when reproducing. One argument in favor of this randomness is that it surfaces those side effects and allows us to identify and clean them. In practice, that doesn't work great: - Being side effects, they are often hard to identify and are a huge time cost for contributors to debug. Our ever increasing list of flake issues is a good testimony of that. - There are not always easy to clean up [1]. - There are often not representative of bugs likely to affect users. Our users rarely restart Cilium 50 times in the span of an hour with different configurations. In the end, our tests were not meant to uncover those side effects and they are therefore inadequate at exposing them properly. If we want to identify those side effects, then we should have tests specifically for that. Other, non-ginkgo CI jobs run their tests in a fixed order [2] or on completely independent clusters [3]. This commit therefore fixes the order of the ginkgo tests. To that end, a ginkgo seed was selected using $SRANDOM and will be the same for all subsequent runs. 1 - cilium#17459 2 - cilium/cilium-cli#558 3 - https://github.com/cilium/cilium/blob/main/.github/workflows/conformance-datapath.yaml Signed-off-by: Paul Chaignon <paul@cilium.io>
pchaigno
force-pushed
the
ginkgo-fix-test-order
branch
from
1624dd1
to
64c7cd0
Compare
|
/test-vagrant |
nebril
approved these changes
Apr 21, 2023
This was referenced Apr 24, 2023
pchaigno
added
the
needs-backport/1.13
|
Marking for backport to v1.13 as #20723 is happening there and this fixes it. |
sayboras
added
backport-pending/1.13
maintainer-s-little-helper
bot
moved this from Needs backport from main
to Backport pending to v1.13
in 1.13.3
sayboras
added
backport-done/1.13
maintainer-s-little-helper
bot
moved this from Backport pending to v1.13
to Backport done to v1.13
in 1.13.3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The ginkgo tests are currently executed in a different order for each run in our CI. Unfortunately, the order in which we run the tests often matters and can impact the test results. That's because there are a lot of side effects (leftover Linux state, leftover Cilium conntrack entries, leftover Kubernetes state, etc.) that are hard to control for.
This undeterminism facilities the introduction of flakes: a first pull request introduces a flake, but it doesn't show up until a later pull request runs the tests in a particular order.
The random order also makes flake a bit order to reproduce locally, though that can easily be worked around by passing the proper seed to ginkgo when reproducing.
One argument in favor of this randomness is that it surfaces those side effects and allows us to identify and clean them. In practice, that doesn't work great:
Other, non-ginkgo CI jobs run their tests in a fixed order [2] or on completely independent clusters [3].
This pull request therefore fixes the order of the ginkgo tests. To that end, a ginkgo seed was selected using $SRANDOM and will be the same for all subsequent runs.