Test de-flaking & bug fixes #7377

JoshVanL · 2024-01-14T00:12:35Z

This PR removes the majority of flakes in our integration and unit tests. The changes include:

Increasing retry timeouts for actor related integration test clients.
Allowing for gRPC server stopped errors in placement_test.go.
Fixing a race condition in the binding send unit test.
Fixing a panic race condition in the sentry JWK validator but ensuring the cache is warm before validating tokens.
Fixing race condition in pluggable component int tests by ensuring the pluggable gRPC server is initialized before running.
Preventing an test assertion panic if the metadata server is not ready when called in int tests.
Cosmetic: Integration tests are now run in alphabetical order by name, and the execution time is outputted.
Fix panic race in actors/healthz/initerror.
Fix panic race in actors/http/ttl.
Fix test assertion panic in reminders/basic.
Fix test assertion panic in reminders/serialisation.
Update all metadata server int test clients to expect the newly always available workflow component, introduced in Configure alternate backend for Dapr workflow #7283
JSON marshal all file name strings to they are compatible with windows runners.
Skip pluggable tests on windows as they rely on unix sockets which are not available yet for windows.
Make placement apiLevel int tests retry connection with placement.
Updates kit to use PR to fix deadlock in fswatcher.

Eventuallys Signed-off-by: joshvanl <me@joshvanl.dev>

codecov · 2024-01-14T00:22:39Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ded0ace) 0.00% compared to head (29e6dc6) 62.17%.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #7377       +/-   ##
===========================================
+ Coverage        0   62.17%   +62.17%     
===========================================
  Files           0      239      +239     
  Lines           0    21942    +21942     
===========================================
+ Hits            0    13643    +13643     
- Misses          0     7166     +7166     
- Partials        0     1133     +1133

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: joshvanl <me@joshvanl.dev>

go.mod

Signed-off-by: joshvanl <me@joshvanl.dev>

mukundansundar · 2024-01-15T18:13:58Z

/ok-to-test

mukundansundar · 2024-01-15T18:14:05Z

/test-version-skew

dapr-bot · 2024-01-15T18:14:21Z

Dapr E2E test

🔗 Link to Action run

Commit ref: fc56151

✅ Build succeeded for linux/amd64

Image tag: dapre2e5d6e4d80eal
Test image tag: dapre2e5d6e4d80eal

✅ Infrastructure deployed

Cluster	Resource group name	Azure region
Linux	`Dapr-E2E-dapre2e5d6e4d80eal`	westus3
Windows	`Dapr-E2E-dapre2e5d6e4d80eaw`	westus3
Linux/arm64	`Dapr-E2E-dapre2e5d6e4d80eala`	eastus

✅ Build succeeded for windows/amd64

Image tag: dapre2e5d6e4d80eaw
Test image tag: dapre2e5d6e4d80eaw

✅ Tests succeeded on windows/amd64

Image tag: dapre2e5d6e4d80eaw
Test image tag: dapre2e5d6e4d80eaw

❌ Tests failed on linux/amd64

Please check the logs for details on the error.

dapr-bot · 2024-01-15T18:14:29Z

Dapr Version Skew integration test (dapr-sidecar-master - 1.12.3)

🔗 Link to Action run

Commit ref: fc56151

✅ Version Skew tests passed

dapr-bot · 2024-01-15T18:14:30Z

Dapr Version Skew integration test (control-plane-master - 1.12.3)

🔗 Link to Action run

Commit ref: fc56151

✅ Version Skew tests passed

dapr-bot · 2024-01-15T18:14:32Z

Dapr Version Skew e2e test (dapr-sidecar-master - 1.12.3)

🔗 Link to Action run

Commit ref: fc56151

❌ Version Skew tests failed

Please check the logs for details on the error.

dapr-bot · 2024-01-15T18:14:33Z

Dapr Version Skew e2e test (control-plane-master - 1.12.3)

🔗 Link to Action run

Commit ref: fc56151

✅ Version Skew tests passed

Signed-off-by: joshvanl <me@joshvanl.dev>

yaron2

Great PR

* Adds test execution time to integration suite, don't panic on some Eventuallys Signed-off-by: joshvanl <me@joshvanl.dev> * Prevent GetMetaComponent from panicing tests if it fails Signed-off-by: joshvanl <me@joshvanl.dev> * Assert client.GetMetadata gRPC response in crypto hot reload tests Signed-off-by: joshvanl <me@joshvanl.dev> * Assert error inside eventually when calling reminders invoke actor Signed-off-by: joshvanl <me@joshvanl.dev> * Increase timeout invoking actor in reminder int tests Signed-off-by: joshvanl <me@joshvanl.dev> * Use events batcher update with events/queue backend Signed-off-by: joshvanl <me@joshvanl.dev> * go mod tidy Signed-off-by: joshvanl <me@joshvanl.dev> * Import efficiency of some tests Signed-off-by: joshvanl <me@joshvanl.dev> * Disable API level tests Signed-off-by: joshvanl <me@joshvanl.dev> * Return placement apilevel tests Signed-off-by: joshvanl <me@joshvanl.dev> * Disable actors tests Signed-off-by: joshvanl <me@joshvanl.dev> * Update queue and batcher for generic key Signed-off-by: joshvanl <me@joshvanl.dev> * Reenable actors int tests and disable hotreload int test Signed-off-by: joshvanl <me@joshvanl.dev> * pkg/placement/placement_test.go: ignore gRPC server shutdown error Signed-off-by: joshvanl <me@joshvanl.dev> * int: enable hotreload tests. Disable hotreload operator tests Signed-off-by: joshvanl <me@joshvanl.dev> * Increase timeout on sentry healthz setup Signed-off-by: joshvanl <me@joshvanl.dev> * Fix race condition in sentry jwks validator Signed-off-by: joshvanl <me@joshvanl.dev> * Reenable subset of hotreload operator tests Signed-off-by: joshvanl <me@joshvanl.dev> * Fix binding send unit test Signed-off-by: joshvanl <me@joshvanl.dev> * Add ping to ensure pluggable state store is available in Run Signed-off-by: joshvanl <me@joshvanl.dev> * Re enable all hotreload operator tests except informer Signed-off-by: joshvanl <me@joshvanl.dev> * Remove secret and state operator hot reload tests Signed-off-by: joshvanl <me@joshvanl.dev> * Skip errors.go tests on windows which rely on unix sockets Signed-off-by: joshvanl <me@joshvanl.dev> * JSON string directory values Signed-off-by: joshvanl <me@joshvanl.dev> * Restore lost hotreload/operator tests with fixed file name escaping Signed-off-by: joshvanl <me@joshvanl.dev> * Linting Signed-off-by: joshvanl <me@joshvanl.dev> * Make httpendpoints windows string check more generic Signed-off-by: joshvanl <me@joshvanl.dev> * Fix metadata related tests because we now have a permanent dapr workflow component Signed-off-by: joshvanl <me@joshvanl.dev> * Linting Signed-off-by: joshvanl <me@joshvanl.dev> * Adds wait for state and secret component to be loaded in fuzz tests Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go modules with kit main Signed-off-by: joshvanl <me@joshvanl.dev> * Adds connection retry for windows mtls kubernetes disable Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Cassandra Coyle <cassie@diagrid.io>

Adds test execution time to integration suite, don't panic on some

523479a

Eventuallys Signed-off-by: joshvanl <me@joshvanl.dev>

JoshVanL force-pushed the test-integration-de-flake branch from 25cbb91 to 523479a Compare January 14, 2024 00:14

Prevent GetMetaComponent from panicing tests if it fails

fdf146d

Signed-off-by: joshvanl <me@joshvanl.dev>

JoshVanL force-pushed the test-integration-de-flake branch from f5e5707 to fdf146d Compare January 14, 2024 00:51

JoshVanL added 7 commits January 14, 2024 01:28

Assert client.GetMetadata gRPC response in crypto hot reload tests

e9f089f

Signed-off-by: joshvanl <me@joshvanl.dev>

Assert error inside eventually when calling reminders invoke actor

fc74cd5

Signed-off-by: joshvanl <me@joshvanl.dev>

Increase timeout invoking actor in reminder int tests

b284ac7

Signed-off-by: joshvanl <me@joshvanl.dev>

Use events batcher update with events/queue backend

c0489d3

Signed-off-by: joshvanl <me@joshvanl.dev>

go mod tidy

d079d47

Signed-off-by: joshvanl <me@joshvanl.dev>

Import efficiency of some tests

3f02b9b

Signed-off-by: joshvanl <me@joshvanl.dev>

Disable API level tests

5ca1251

Signed-off-by: joshvanl <me@joshvanl.dev>

JoshVanL force-pushed the test-integration-de-flake branch from cbf6ba1 to 5ca1251 Compare January 14, 2024 16:06

JoshVanL added 17 commits January 14, 2024 16:54

Return placement apilevel tests

9181442

Signed-off-by: joshvanl <me@joshvanl.dev>

Disable actors tests

d739dce

Signed-off-by: joshvanl <me@joshvanl.dev>

Update queue and batcher for generic key

214350f

Signed-off-by: joshvanl <me@joshvanl.dev>

Reenable actors int tests and disable hotreload int test

396c2de

Signed-off-by: joshvanl <me@joshvanl.dev>

pkg/placement/placement_test.go: ignore gRPC server shutdown error

60f8cb2

Signed-off-by: joshvanl <me@joshvanl.dev>

int: enable hotreload tests. Disable hotreload operator tests

498790a

Signed-off-by: joshvanl <me@joshvanl.dev>

Increase timeout on sentry healthz setup

97735e6

Signed-off-by: joshvanl <me@joshvanl.dev>

Fix race condition in sentry jwks validator

286aa7c

Signed-off-by: joshvanl <me@joshvanl.dev>

Reenable subset of hotreload operator tests

f405e1b

Signed-off-by: joshvanl <me@joshvanl.dev>

Fix binding send unit test

a1e2436

Signed-off-by: joshvanl <me@joshvanl.dev>

Add ping to ensure pluggable state store is available in Run

adfe113

Signed-off-by: joshvanl <me@joshvanl.dev>

Re enable all hotreload operator tests except informer

156b81d

Signed-off-by: joshvanl <me@joshvanl.dev>

Remove secret and state operator hot reload tests

f658675

Signed-off-by: joshvanl <me@joshvanl.dev>

Skip errors.go tests on windows which rely on unix sockets

d37a671

Signed-off-by: joshvanl <me@joshvanl.dev>

JSON string directory values

f0e5ef8

Signed-off-by: joshvanl <me@joshvanl.dev>

Restore lost hotreload/operator tests with fixed file name escaping

8715a4e

Signed-off-by: joshvanl <me@joshvanl.dev>

Linting

44e9d31

Signed-off-by: joshvanl <me@joshvanl.dev>

JoshVanL added 2 commits January 15, 2024 15:59

Linting

e228f17

Signed-off-by: joshvanl <me@joshvanl.dev>

Adds wait for state and secret component to be loaded in fuzz tests

9a1af9e

Signed-off-by: joshvanl <me@joshvanl.dev>

JoshVanL changed the title ~~Integration: de-flake~~ Test de-flaking Jan 15, 2024

JoshVanL changed the title ~~Test de-flaking~~ Test de-flaking & bug fixes Jan 15, 2024

JoshVanL added do-not-merge autoupdate DaprBot will keep the Pull Request up to date with master branch labels Jan 15, 2024

JoshVanL marked this pull request as ready for review January 15, 2024 17:03

JoshVanL requested review from a team as code owners January 15, 2024 17:03

artursouza previously approved these changes Jan 15, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

Updates go modules with kit main

fc56151

Signed-off-by: joshvanl <me@joshvanl.dev>

JoshVanL dismissed artursouza’s stale review via fc56151 January 15, 2024 17:23

JoshVanL removed the do-not-merge label Jan 15, 2024

artursouza previously approved these changes Jan 15, 2024

View reviewed changes

mukundansundar previously approved these changes Jan 15, 2024

View reviewed changes

Adds connection retry for windows mtls kubernetes disable

29e6dc6

Signed-off-by: joshvanl <me@joshvanl.dev>

JoshVanL dismissed stale reviews from mukundansundar and artursouza via 29e6dc6 January 15, 2024 18:16

elena-kolevska approved these changes Jan 15, 2024

View reviewed changes

yaron2 approved these changes Jan 15, 2024

View reviewed changes

yaron2 merged commit 8d01d77 into dapr:master Jan 15, 2024
31 of 32 checks passed

JoshVanL added this to the v1.13 milestone Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test de-flaking & bug fixes #7377

Test de-flaking & bug fixes #7377

JoshVanL commented Jan 14, 2024 •

edited

codecov bot commented Jan 14, 2024 •

edited

mukundansundar commented Jan 15, 2024

mukundansundar commented Jan 15, 2024

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

yaron2 left a comment

Test de-flaking & bug fixes #7377

Test de-flaking & bug fixes #7377

Conversation

JoshVanL commented Jan 14, 2024 • edited

codecov bot commented Jan 14, 2024 • edited

Codecov Report

mukundansundar commented Jan 15, 2024

mukundansundar commented Jan 15, 2024

dapr-bot commented Jan 15, 2024 • edited

Dapr E2E test

✅ Build succeeded for linux/amd64

✅ Infrastructure deployed

✅ Build succeeded for windows/amd64

✅ Tests succeeded on windows/amd64

❌ Tests failed on linux/amd64

dapr-bot commented Jan 15, 2024 • edited

Dapr Version Skew integration test (dapr-sidecar-master - 1.12.3)

✅ Version Skew tests passed

dapr-bot commented Jan 15, 2024 • edited

Dapr Version Skew integration test (control-plane-master - 1.12.3)

✅ Version Skew tests passed

dapr-bot commented Jan 15, 2024 • edited

Dapr Version Skew e2e test (dapr-sidecar-master - 1.12.3)

❌ Version Skew tests failed

dapr-bot commented Jan 15, 2024 • edited

Dapr Version Skew e2e test (control-plane-master - 1.12.3)

✅ Version Skew tests passed

yaron2 left a comment

Choose a reason for hiding this comment

JoshVanL commented Jan 14, 2024 •

edited

codecov bot commented Jan 14, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited

dapr-bot commented Jan 15, 2024 •

edited