Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concurrent map writes for component providers #11160

Closed
iwahbe opened this issue Oct 26, 2022 · 1 comment
Closed

Concurrent map writes for component providers #11160

iwahbe opened this issue Oct 26, 2022 · 1 comment
Labels
impact/panic This bug represents a panic or unexpected crash impact/regression Something that used to work, but is now broken kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer resolution/duplicate This issue is a duplicate of another issue

Comments

@iwahbe
Copy link
Member

iwahbe commented Oct 26, 2022

What happened?

We saw panics on concurrent component provider creates.

Steps to reproduce

Unknown

Expected Behavior

No panics

Actual Behavior

Stack trace
Stack trace is partial since the whole stack trace is too big for GitHub. 
fatal error: concurrent map writes

goroutine 11308 [running]:
github.com/pulumi/pulumi/pkg/v3/resource/deploy.(*resmon).RegisterResource(0xc00592f080, {0xc00d981858?, 0x7d0875?}, 0xc000e5ea80)
        /home/runner/work/pulumi/pulumi/pkg/resource/deploy/source_eval.go:1204 +0x2b07
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceMonitor_RegisterResource_Handler.func1({0x28706f8, 0xc003ab0540}, {0x2267120?, 0xc000e5ea80})
        /home/runner/work/pulumi/pulumi/sdk/proto/go/resource.pb.go:1702 +0x78
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x28706f8, 0xc003ab0480}, {0x2267120, 0xc000e5ea80}, 0xc003d02100, 0xc000db23a8)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3f9
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceMonitor_RegisterResource_Handler({0x219ac40?, 0xc00592f080}, {0x28706f8, 0xc003ab0480}, 0xc000a78150, 0xc003368900)
        /home/runner/work/pulumi/pulumi/sdk/proto/go/resource.pb.go:1704 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc005a32b40, {0x287a960, 0xc005cffd40}, 0xc00778a360, 0xc00100bb30, 0x3861ce0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1301 +0xb2b
google.golang.org/grpc.(*Server).handleStream(0xc005a32b40, {0x287a960, 0xc005cffd40}, 0xc00778a360, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1642 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:936 +0x28a

goroutine 1 [select, 1 minutes]:
github.com/pulumi/pulumi/pkg/v3/engine.(*deployment).run(0xc001ef8dc0, 0xc004f0f4c0, {0x2876698, 0xc000e5f380}, 0x0?, 0x0)
        /home/runner/work/pulumi/pulumi/pkg/engine/deployment.go:288 +0x43e
github.com/pulumi/pulumi/pkg/v3/engine.update(_, _, {{{0x38b1670, 0x0, 0x0}, {0x0, 0x0, 0x0}, 0x7fffffff, 0x0, ...}, ...}, ...)
        /home/runner/work/pulumi/pulumi/pkg/engine/update.go:457 +0x4c7
github.com/pulumi/pulumi/pkg/v3/engine.Update({0x286d020, 0xc00463fef0}, 0xc004f0f4c0, {{0x38b1670, 0x0, 0x0}, {0x0, 0x0, 0x0}, 0x7fffffff, ...}, ...)
        /home/runner/work/pulumi/pulumi/pkg/engine/update.go:193 +0x670
github.com/pulumi/pulumi/pkg/v3/backend/httpstate.(*cloudBackend).runEngineAction(_, {_, _}, {_, _}, {_, _}, {0xc000531d60, {0xc001b68600, 0x15}, ...}, ...)
        /home/runner/work/pulumi/pulumi/pkg/backend/httpstate/backend.go:1166 +0x88f
github.com/pulumi/pulumi/pkg/v3/backend/httpstate.(*cloudBackend).apply(_, {_, _}, {_, _}, {_, _}, {0xc000531d60, {0xc001b68600, 0x15}, ...}, ...)
        /home/runner/work/pulumi/pulumi/pkg/backend/httpstate/backend.go:1073 +0x41f
github.com/pulumi/pulumi/pkg/v3/backend.PreviewThenPromptThenExecute({_, _}, {_, _}, {_, _}, {0xc000531d60, {0xc001b68600, 0x15}, {0x0, ...}, ...}, ...)
        /home/runner/work/pulumi/pulumi/pkg/backend/apply.go:238 +0x232
github.com/pulumi/pulumi/pkg/v3/backend/httpstate.(*cloudBackend).Update(_, {_, _}, {_, _}, {0xc000531d60, {0xc001b68600, 0x15}, {0x0, 0x0, ...}, ...})
        /home/runner/work/pulumi/pulumi/pkg/backend/httpstate/backend.go:956 +0xc5
github.com/pulumi/pulumi/pkg/v3/backend.UpdateStack({_, _}, {_, _}, {0xc000531d60, {0xc001b68600, 0x15}, {0x0, 0x0, 0x0}, ...})
        /home/runner/work/pulumi/pulumi/pkg/backend/stack.go:91 +0xba
github.com/pulumi/pulumi/pkg/v3/backend/httpstate.(*cloudStack).Update(_, {_, _}, {0xc000531d60, {0xc001b68600, 0x15}, {0x0, 0x0, 0x0}, 0xc001b661e0, ...})
        /home/runner/work/pulumi/pulumi/pkg/backend/httpstate/stack.go:152 +0x88
main.newUpCmd.func1({_, _}, {{{0x38b1670, 0x0, 0x0}, {0x0, 0x0, 0x0}, 0x7fffffff, 0x0, ...}, ...})
        /home/runner/work/pulumi/pulumi/pkg/cmd/pulumi/up.go:198 +0xfb7
main.newUpCmd.func3(0x0?, {0xc000795290, 0x0, 0x0?})
        /home/runner/work/pulumi/pulumi/pkg/cmd/pulumi/up.go:488 +0x665
github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil.RunResultFunc.func1(0xc00157cc80?, {0xc000795290, 0x0, 0x1})
        /home/runner/work/pulumi/pulumi/sdk/go/common/util/cmdutil/exit.go:112 +0x4f
github.com/spf13/cobra.(*Command).execute(0xc00157cc80, {0xc000795280, 0x1, 0x1})
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0xc0013caf00)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /home/runner/work/pulumi/pulumi/pkg/cmd/pulumi/main.go:48 +0x45

goroutine 6 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
        /home/runner/go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:882 +0x6a
created by github.com/golang/glog.init.0
        /home/runner/go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:410 +0x1bf

goroutine 934 [select, 1 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000bbfc20, 0x1)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/internal/transport/controlbuf.go:408 +0x115
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0015890e0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/internal/transport/controlbuf.go:535 +0x85
google.golang.org/grpc/internal/transport.NewServerTransport.func2()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/internal/transport/http2_server.go:331 +0xce
created by google.golang.org/grpc/internal/transport.NewServerTransport
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/internal/transport/http2_server.go:328 +0x1913

goroutine 127 [select]:
github.com/pulumi/pulumi/pkg/v3/backend/httpstate.(*tokenSource).handleRequests(0xc001eec3c0, {0x2870688, 0xc00005c1c0}, {0xc001d94fc0, 0x217}, {0xc00138efd0?, 0x7856aa?, 0x3880d20?}, 0x45d964b800, 0xc001eee770)
        /home/runner/work/pulumi/pulumi/pkg/backend/httpstate/token_source.go:112 +0x270
created by github.com/pulumi/pulumi/pkg/v3/backend/httpstate.newTokenSource
        /home/runner/work/pulumi/pulumi/pkg/backend/httpstate/token_source.go:57 +0x1ca

Output of pulumi about

CLI          
Version      3.43.1
Go Version   go1.19.1
Go Compiler  gc

Additional context

Customer reported

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@iwahbe iwahbe added kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed labels Oct 26, 2022
@iwahbe
Copy link
Member Author

iwahbe commented Oct 26, 2022

Fixed #11151.

@iwahbe iwahbe closed this as completed Oct 26, 2022
@iwahbe iwahbe added impact/panic This bug represents a panic or unexpected crash impact/regression Something that used to work, but is now broken resolution/duplicate This issue is a duplicate of another issue and removed resolution/fixed This issue was fixed labels Oct 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
impact/panic This bug represents a panic or unexpected crash impact/regression Something that used to work, but is now broken kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer resolution/duplicate This issue is a duplicate of another issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@iwahbe