test/integration: Avoid data race from FileMutex #12049

abhinav · 2023-02-02T20:54:27Z

integration_util_test contains a helper function synchronouslyDo
which attempts to run an operation with a file lock with a timeout,
in a blocking manner.
This implementation has a couple issues.

First, there's a data race between Lock() and Unlock() of the mutex:
the function defers an unlock regardless of whether a lock was acquired,
and if the timing is right, it causes a data race in FileMutex
on reading and writing the fsunlock field.

WARNING: DATA RACE
Read at 0x00c000388040 by goroutine 16:
  github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Unlock()
      /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:64 +0x3e
  github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func1()
      /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:263 +0x39
  [..]

Previous write at 0x00c000388040 by goroutine 17:
  github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Lock()
      /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:55 +0x72
  github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func2()
      /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:269 +0x4e

This data race is not expected because the contract for FileMutex is
that the Unlock method should only be called after Lock,
typicaly from the same goroutine --- synchronouslyDo does not do this.

Secondly, synchronouslyDo has a minor bug:
it will run the function eventually when the lock has been acquired
even if the timeout has expirted and the test has failed by then.

Resolve these issues by making the following changes:

use a context to track the timeout
defer an unlock only if a lock was successfully acquired
run the operation only if we still have time to run it

Includes a failing test case.

abhinav · 2023-02-02T20:54:39Z

Current dependencies on/for this PR:

master
- PR ci: Use macos-latest #12066
  - PR test/integration: Avoid data race from FileMutex #12049 👈
  - PR test/integration: Don't panic during setup #12068
    - PR test/integration: Capture component setup output #12069

This comment was auto-generated by Graphite.

pulumi-bot · 2023-02-02T20:55:03Z

Changelog

[uncommitted] (2023-02-03)

Frassle · 2023-02-02T21:45:07Z

tests/integration/integration_util_test.go

+
+		// ctx.Err will be non-nil when the context finishes
+		// either because it timed out or because it got canceled.
+		for ctx.Err() == nil {
 			if err := mutex.Lock(); err != nil {
 				time.Sleep(1 * time.Second)


No need to change in this changeset but I wonder if this sleep loop could be improved to speed up tests? If a lot of these ops locked by synchronouslyDo are ms might be worth tuning.

Ah answered my own question, it's only used for building components so 1s is probably on par with the script being ran.

abhinav · 2023-02-02T22:00:37Z

bors r+

12025: [sdks/go] Delegate alias computation to the engine r=Zaid-Ajaj a=Zaid-Ajaj Fixes #11066 Addresses #11697 Credit to `@abhinav` for making aliases unit-testable by intercepting `RegisterResource` calls. > I did change the test slightly so that it either checks for `AliasURNs: []string` or `Aliases: []*pulumirpc.Alias` because I've made it such that one of them is `nil` depending on `supportsAliasSpecs` ## Checklist  - [x] I have added tests that prove my fix is effective or that my feature works  - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  12046: Use 'errors' not 'pkg/errors' in go codegen r=Frassle a=Frassle  # Description  Continue clean up of our use of pkg/errors. This changes our Go code generator to stop using it, there's still a few places in the sdk using it so go.mods will still reference it. Looks like the only thing the code generator used "pkg/errors" for was `New` which is also on "errors". ## Checklist  - [ ] I have added tests that prove my fix is effective or that my feature works - Covered by existing tests  - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  12049: test/integration: Avoid data race from FileMutex r=abhinav a=abhinav integration_util_test contains a helper function synchronouslyDo which attempts to run an operation with a file lock with a timeout, in a blocking manner. This implementation has a couple issues. First, there's a data race between Lock() and Unlock() of the mutex: the function defers an unlock regardless of whether a lock was acquired, and if the timing is right, it causes a data race in FileMutex on reading and writing the fsunlock field. ``` WARNING: DATA RACE Read at 0x00c000388040 by goroutine 16: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Unlock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:64 +0x3e github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func1() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:263 +0x39 [..] Previous write at 0x00c000388040 by goroutine 17: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Lock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:55 +0x72 github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func2() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:269 +0x4e ``` This data race is not expected because the contract for FileMutex is that the Unlock method should only be called after Lock, typicaly from the same goroutine --- synchronouslyDo does not do this. Secondly, synchronouslyDo has a minor bug: it will run the function *eventually* when the lock has been acquired even if the timeout has expirted and the test has failed by then. Resolve these issues by making the following changes: - use a context to track the timeout - defer an unlock only if a lock was successfully acquired - run the operation only if we still have time to run it Includes a failing test case. Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com> Co-authored-by: Fraser Waters <fraser@pulumi.com> Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>

bors · 2023-02-02T23:30:08Z

Build failed (retrying...):

bors-ok

12046: Use 'errors' not 'pkg/errors' in go codegen r=Frassle a=Frassle  # Description  Continue clean up of our use of pkg/errors. This changes our Go code generator to stop using it, there's still a few places in the sdk using it so go.mods will still reference it. Looks like the only thing the code generator used "pkg/errors" for was `New` which is also on "errors". ## Checklist  - [ ] I have added tests that prove my fix is effective or that my feature works - Covered by existing tests  - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  12049: test/integration: Avoid data race from FileMutex r=abhinav a=abhinav integration_util_test contains a helper function synchronouslyDo which attempts to run an operation with a file lock with a timeout, in a blocking manner. This implementation has a couple issues. First, there's a data race between Lock() and Unlock() of the mutex: the function defers an unlock regardless of whether a lock was acquired, and if the timing is right, it causes a data race in FileMutex on reading and writing the fsunlock field. ``` WARNING: DATA RACE Read at 0x00c000388040 by goroutine 16: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Unlock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:64 +0x3e github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func1() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:263 +0x39 [..] Previous write at 0x00c000388040 by goroutine 17: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Lock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:55 +0x72 github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func2() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:269 +0x4e ``` This data race is not expected because the contract for FileMutex is that the Unlock method should only be called after Lock, typicaly from the same goroutine --- synchronouslyDo does not do this. Secondly, synchronouslyDo has a minor bug: it will run the function *eventually* when the lock has been acquired even if the timeout has expirted and the test has failed by then. Resolve these issues by making the following changes: - use a context to track the timeout - defer an unlock only if a lock was successfully acquired - run the operation only if we still have time to run it Includes a failing test case. Co-authored-by: Fraser Waters <fraser@pulumi.com> Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>

abhinav · 2023-02-03T00:10:45Z

bors cancel

bors · 2023-02-03T00:10:47Z

Canceled.

Cherry-picks #12049 into the test PR since that also includes some fixes for how component setup works.

Frassle · 2023-02-03T09:18:01Z

bors merge

12049: test/integration: Avoid data race from FileMutex r=Frassle a=abhinav integration_util_test contains a helper function synchronouslyDo which attempts to run an operation with a file lock with a timeout, in a blocking manner. This implementation has a couple issues. First, there's a data race between Lock() and Unlock() of the mutex: the function defers an unlock regardless of whether a lock was acquired, and if the timing is right, it causes a data race in FileMutex on reading and writing the fsunlock field. ``` WARNING: DATA RACE Read at 0x00c000388040 by goroutine 16: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Unlock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:64 +0x3e github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func1() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:263 +0x39 [..] Previous write at 0x00c000388040 by goroutine 17: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Lock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:55 +0x72 github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func2() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:269 +0x4e ``` This data race is not expected because the contract for FileMutex is that the Unlock method should only be called after Lock, typicaly from the same goroutine --- synchronouslyDo does not do this. Secondly, synchronouslyDo has a minor bug: it will run the function *eventually* when the lock has been acquired even if the timeout has expirted and the test has failed by then. Resolve these issues by making the following changes: - use a context to track the timeout - defer an unlock only if a lock was successfully acquired - run the operation only if we still have time to run it Includes a failing test case. Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>

bors · 2023-02-03T09:44:36Z

Build failed:

bors-ok

Frassle · 2023-02-03T09:47:33Z

Error: ../../../../go/pkg/mod/gocloud.dev@v0.27.0/secrets/azurekeyvault/akv.go:49:2: github.com/Azure/azure-sdk-for-go@v66.0.0+incompatible: read "https:/proxy.golang.org/@v/v66.0.0+incompatible.zip": http2: server sent GOAWAY and closed the connection; LastStreamID=7, ErrCode=NO_ERROR, debug="server_shutting_down"
Go cache building failed

bors retry

12049: test/integration: Avoid data race from FileMutex r=Frassle a=abhinav integration_util_test contains a helper function synchronouslyDo which attempts to run an operation with a file lock with a timeout, in a blocking manner. This implementation has a couple issues. First, there's a data race between Lock() and Unlock() of the mutex: the function defers an unlock regardless of whether a lock was acquired, and if the timing is right, it causes a data race in FileMutex on reading and writing the fsunlock field. ``` WARNING: DATA RACE Read at 0x00c000388040 by goroutine 16: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Unlock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:64 +0x3e github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func1() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:263 +0x39 [..] Previous write at 0x00c000388040 by goroutine 17: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Lock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:55 +0x72 github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func2() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:269 +0x4e ``` This data race is not expected because the contract for FileMutex is that the Unlock method should only be called after Lock, typicaly from the same goroutine --- synchronouslyDo does not do this. Secondly, synchronouslyDo has a minor bug: it will run the function *eventually* when the lock has been acquired even if the timeout has expirted and the test has failed by then. Resolve these issues by making the following changes: - use a context to track the timeout - defer an unlock only if a lock was successfully acquired - run the operation only if we still have time to run it Includes a failing test case. Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>

Frassle · 2023-02-03T10:30:09Z

tests/integration/integration_util_test.go

 	}()

 	select {
-	case <-time.After(timeout):
+	case <-ctx.Done():


Can we get rid of this select and just wait for lockWait.
My worry is that we could start running fn() in the goroutine, then the timeout hits and this select fires, and we just leave the goroutine running and don't get to see it's result.

Guess it means lockWait needs to be a bool of true/false for it it timed out or not.

bors · 2023-02-03T10:54:17Z

Build failed:

bors-ok

Frassle · 2023-02-03T18:49:11Z

I think given this is an improvement over what's currently in, and is ready to go we should merge it. But I'm going to take a look to see if we can just delete filelocks from our system entirely. They're used in two places:

Here where I think we can actually just use process mutex because we don't run test processes in parallel.
In the plugin code, which even with a lock I'm pretty sure is racy.

abhinav · 2023-02-03T19:02:05Z

Ack.
Superseded by #12065

abhinav · 2023-02-03T20:08:40Z

Never mind, #12065 was incorrect.
We do run external binaries in parallel, so we need the file lock.

integration_util_test contains a helper function synchronouslyDo which attempts to run an operation with a file lock with a timeout, in a blocking manner. This implementation has a couple issues. First, there's a data race between Lock() and Unlock() of the mutex: the function defers an unlock regardless of whether a lock was acquired, and if the timing is right, it causes a data race in FileMutex on reading and writing the fsunlock field. ``` WARNING: DATA RACE Read at 0x00c000388040 by goroutine 16: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Unlock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:64 +0x3e github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func1() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:263 +0x39 [..] Previous write at 0x00c000388040 by goroutine 17: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Lock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:55 +0x72 github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func2() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:269 +0x4e ``` This data race is not expected because the contract for FileMutex is that the Unlock method should only be called after Lock, typicaly from the same goroutine --- synchronouslyDo does not do this. Secondly, synchronouslyDo has a minor bug: it will run the function *eventually* when the lock has been acquired even if the timeout has expirted and the test has failed by then. Resolve these issues by making the following changes: - use a context to track the timeout - defer an unlock only if a lock was successfully acquired - run the operation only if we still have time to run it Includes a previously failing test case.

abhinav · 2023-02-03T22:15:36Z

bors r+

12025: [sdks/go] Delegate alias computation to the engine r=abhinav a=Zaid-Ajaj Fixes #11066 Addresses #11697 Credit to `@abhinav` for making aliases unit-testable by intercepting `RegisterResource` calls. > I did change the test slightly so that it either checks for `AliasURNs: []string` or `Aliases: []*pulumirpc.Alias` because I've made it such that one of them is `nil` depending on `supportsAliasSpecs` ## Checklist  - [x] I have added tests that prove my fix is effective or that my feature works  - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  12028: Require linting before running unit, integ, and smoke tests. r=abhinav a=RobbieMcKinstry  # Description **Update:** With #12031 linting runs in about a minute. Linting should be an extremely low-watermark requirement for evaluating build health. Blocking on it allows use to reduce the number of concurrent runners who are canceled early. **Trade-offs:** * This should delay CI time by the amount of time it takes to lint: _e.g._ CI will be ~5 minutes slower on the happy path. * When a job is queued that fails a lint check, fewer runners will be soaked up just to fail lint checks. This will decrease the overall queue time across all builds. * Ultimately, we're trading slower happy-path builds for smarter build scheduling. * We can mitigate the linting bottleneck by speeding up the linting process (#12023).  This PR supports but isn't sufficient for #12019 ## Checklist **This PR is intended to impact CI only, and thus does not justify a CHANGELOG entry or a test.**  - [ ] I have added tests that prove my fix is effective or that my feature works  - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  12043: sdk/go: Don't store DependsOn in a lossy form r=abhinav a=abhinav The `DependsOn` and `DependsOnInputs` resource options store their captured information on the `resourceOptions` struct in a lossy format: they store function references. This makes it impossible to go back to the original lists of resources or resource array inputs for use cases like #11698. As a step towards making this possible, replace the stored closures with interfaces. The implementations in the first commit are a drop-in replacement for the prior behavior with no logic changes whatsoever. The second commit makes a minor optimization: it adds URNs to the same set instead of constantly allocating new sets and combining them afterwards. Refs #11698 12046: Use 'errors' not 'pkg/errors' in go codegen r=abhinav a=Frassle  # Description  Continue clean up of our use of pkg/errors. This changes our Go code generator to stop using it, there's still a few places in the sdk using it so go.mods will still reference it. Looks like the only thing the code generator used "pkg/errors" for was `New` which is also on "errors". ## Checklist  - [ ] I have added tests that prove my fix is effective or that my feature works - Covered by existing tests  - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change  - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version  12047: pkg/errors cleanup for sdk/go/common/resource/config r=abhinav a=Frassle Continuing pkg/errors cleanup. 12049: test/integration: Avoid data race from FileMutex r=abhinav a=abhinav integration_util_test contains a helper function synchronouslyDo which attempts to run an operation with a file lock with a timeout, in a blocking manner. This implementation has a couple issues. First, there's a data race between Lock() and Unlock() of the mutex: the function defers an unlock regardless of whether a lock was acquired, and if the timing is right, it causes a data race in FileMutex on reading and writing the fsunlock field. ``` WARNING: DATA RACE Read at 0x00c000388040 by goroutine 16: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Unlock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:64 +0x3e github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func1() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:263 +0x39 [..] Previous write at 0x00c000388040 by goroutine 17: github.com/pulumi/pulumi/sdk/v3/go/common/util/fsutil.(*FileMutex).Lock() /Users/runner/work/pulumi/pulumi/sdk/go/common/util/fsutil/lock.go:55 +0x72 github.com/pulumi/pulumi/tests/integration.synchronouslyDo.func2() /Users/runner/work/pulumi/pulumi/tests/integration/integration_util_test.go:269 +0x4e ``` This data race is not expected because the contract for FileMutex is that the Unlock method should only be called after Lock, typicaly from the same goroutine --- synchronouslyDo does not do this. Secondly, synchronouslyDo has a minor bug: it will run the function *eventually* when the lock has been acquired even if the timeout has expirted and the test has failed by then. Resolve these issues by making the following changes: - use a context to track the timeout - defer an unlock only if a lock was successfully acquired - run the operation only if we still have time to run it Includes a failing test case. 12068: test/integration: Don't panic during setup r=abhinav a=abhinav Currently, component setup panics if there's an error. This isn't great because when it panics, it fails to notify the outer goroutine that's waiting for setup, which leaves that goroutine waiting for 10 minutes gefore giving up. The net effect of this is that when setup fails for a test, it takes 10 minutes to kill the test even if setup failed within seconds. Fix this by using testify and logging errors right away. Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com> Co-authored-by: Robbie McKinstry <robbie@pulumi.com> Co-authored-by: Abhinav Gupta <abhinav@pulumi.com> Co-authored-by: Fraser Waters <fraser@pulumi.com>

bors · 2023-02-04T00:07:00Z

Build failed (retrying...):

bors-ok

bors · 2023-02-04T01:53:46Z

Build succeeded:

bors-ok

abhinav added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Feb 2, 2023

abhinav requested review from AaronFriel and justinvp February 2, 2023 21:21

Frassle approved these changes Feb 2, 2023

View reviewed changes

abhinav added a commit that referenced this pull request Feb 3, 2023

Cherry pick #12049

2db2a06

Cherry-picks #12049 into the test PR since that also includes some fixes for how component setup works.

Frassle reviewed Feb 3, 2023

View reviewed changes

abhinav closed this Feb 3, 2023

abhinav deleted the abhinav/integration-sync-do-race branch February 3, 2023 19:02

abhinav restored the abhinav/integration-sync-do-race branch February 3, 2023 20:08

abhinav reopened this Feb 3, 2023

abhinav mentioned this pull request Feb 3, 2023

Remove our use of filelocks in component tests #12065

Closed

3 tasks

abhinav changed the base branch from master to abhinav/macos-latest February 3, 2023 21:15

abhinav force-pushed the abhinav/integration-sync-do-race branch from e6588b2 to dd69dfc Compare February 3, 2023 21:15

This was referenced Feb 3, 2023

ci: Use macos-latest #12066

Merged

test/integration: Don't panic during setup #12068

Merged

test/integration: Capture component setup output #12069

Merged

Base automatically changed from abhinav/macos-latest to master February 3, 2023 22:06

bors bot merged commit 7675a4e into master Feb 4, 2023

bors bot deleted the abhinav/integration-sync-do-race branch February 4, 2023 01:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test/integration: Avoid data race from FileMutex #12049

test/integration: Avoid data race from FileMutex #12049

abhinav commented Feb 2, 2023 •

edited

abhinav commented Feb 2, 2023 •

edited

pulumi-bot commented Feb 2, 2023 •

edited

Frassle Feb 2, 2023

Frassle Feb 2, 2023

abhinav commented Feb 2, 2023

bors bot commented Feb 2, 2023

abhinav commented Feb 3, 2023

bors bot commented Feb 3, 2023

Frassle commented Feb 3, 2023

bors bot commented Feb 3, 2023

Frassle commented Feb 3, 2023

Frassle Feb 3, 2023

Frassle Feb 3, 2023

bors bot commented Feb 3, 2023

Frassle commented Feb 3, 2023

abhinav commented Feb 3, 2023

abhinav commented Feb 3, 2023

abhinav commented Feb 3, 2023

bors bot commented Feb 4, 2023

bors bot commented Feb 4, 2023

test/integration: Avoid data race from FileMutex #12049

test/integration: Avoid data race from FileMutex #12049

Conversation

abhinav commented Feb 2, 2023 • edited

abhinav commented Feb 2, 2023 • edited

pulumi-bot commented Feb 2, 2023 • edited

Changelog

[uncommitted] (2023-02-03)

Frassle Feb 2, 2023

Choose a reason for hiding this comment

Frassle Feb 2, 2023

Choose a reason for hiding this comment

abhinav commented Feb 2, 2023

bors bot commented Feb 2, 2023

abhinav commented Feb 3, 2023

bors bot commented Feb 3, 2023

Frassle commented Feb 3, 2023

bors bot commented Feb 3, 2023

Frassle commented Feb 3, 2023

Frassle Feb 3, 2023

Choose a reason for hiding this comment

Frassle Feb 3, 2023

Choose a reason for hiding this comment

bors bot commented Feb 3, 2023

Frassle commented Feb 3, 2023

abhinav commented Feb 3, 2023

abhinav commented Feb 3, 2023

abhinav commented Feb 3, 2023

bors bot commented Feb 4, 2023

bors bot commented Feb 4, 2023

abhinav commented Feb 2, 2023 •

edited

abhinav commented Feb 2, 2023 •

edited

pulumi-bot commented Feb 2, 2023 •

edited