Address a data race issue within identity_store_util::processLocalAlias #13093
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- When loading an existing alias within processLocalAlias we aren't
cloning the object from the memory store. There seems to be a data
race within the function when calling entity.UpsertAlias and
a concurrent invalidation routine.
==================
WARNING: DATA RACE
Read at 0x00c00bd03d08 by goroutine 94:
google.golang.org/protobuf/internal/impl.pointer.Elem()
/go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/pointer_unsafe.go:118 +0x2b3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow()
/go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:76 +0x265
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer()
/go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:56 +0x12a
google.golang.org/protobuf/internal/impl.(*MessageInfo).size()
/go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:40 +0x95
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm()
/go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:33 +0x6c
google.golang.org/protobuf/proto.MarshalOptions.marshal()
/go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:153 +0x1f3
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend()
/go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:122 +0xa5
github.com/golang/protobuf/proto.marshalAppend()
/go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xe4
github.com/golang/protobuf/proto.Marshal()
/go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23 +0x64
github.com/hashicorp/vault/helper/identity.(*Entity).Clone()
/go/src/github.com/hashicorp/vault/helper/identity/identity.go:34 +0x150
github.com/hashicorp/vault/vault.(*IdentityStore).MemDBEntitiesByBucketKeyInTxn()
/go/src/github.com/hashicorp/vault/vault/identity_store_util.go:1214 +0x306
github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate()
/go/src/github.com/hashicorp/vault/vault/identity_store.go:216 +0xd6c
github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate-fm()
/go/src/github.com/hashicorp/vault/vault/identity_store.go:160 +0x6d
github.com/hashicorp/vault/sdk/framework.(*Backend).InvalidateKey()
/go/src/github.com/hashicorp/vault/sdk/framework/backend.go:347 +0x8a
github.com/hashicorp/vault/vault.(*IdentityStore).InvalidateKey()
<autogenerated>:1 +0x7d
github.com/hashicorp/vault/vault.(*Core).asyncInvalidateKey()
/go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:58 +0x390
github.com/hashicorp/vault/vault.(*Core).asyncInvalidateHandler()
/go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:71 +0x9b
github.com/hashicorp/vault/vault.startReplicationEnt·dwrap·453()
/go/src/github.com/hashicorp/vault/vault/replication_util_ent.go:331 +0x71
Previous write at 0x00c00bd03d08 by goroutine 52:
github.com/hashicorp/vault/helper/identity.(*Entity).UpsertAlias()
/go/src/github.com/hashicorp/vault/helper/identity/identity.go:55 +0x271
github.com/hashicorp/vault/vault.(*IdentityStore).processLocalAlias()
/go/src/github.com/hashicorp/vault/vault/identity_store_util.go:720 +0x672
github.com/hashicorp/vault/vault.possiblyForwardEntityCreation()
/go/src/github.com/hashicorp/vault/vault/request_handling_util_ent.go:230 +0x286
github.com/hashicorp/vault/vault.(*Core).handleLoginRequest()
/go/src/github.com/hashicorp/vault/vault/request_handling.go:1345 +0x234a
github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest()
/go/src/github.com/hashicorp/vault/vault/request_handling.go:607 +0x1a11
github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest()
/go/src/github.com/hashicorp/vault/vault/request_handling.go:442 +0x5b5
github.com/hashicorp/vault/vault.(*Core).HandleRequest()
/go/src/github.com/hashicorp/vault/vault/request_handling.go:408 +0xf2
github.com/hashicorp/vault/http.request()
/go/src/github.com/hashicorp/vault/http/handler.go:953 +0xb1
github.com/hashicorp/vault/http.handleLogicalInternal.func1()
/go/src/github.com/hashicorp/vault/http/logical.go:341 +0xca
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
github.com/hashicorp/vault/http.handleRequestForwarding.func1()
/go/src/github.com/hashicorp/vault/http/handler.go:887 +0x4eb
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
net/http.(*ServeMux).ServeHTTP()
/usr/local/go/src/net/http/server.go:2424 +0xc5
github.com/hashicorp/vault/http.wrapHelpHandler.func1()
/go/src/github.com/hashicorp/vault/http/help.go:23 +0x281
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
github.com/hashicorp/vault/http.wrapCORSHandler.func1()
/go/src/github.com/hashicorp/vault/http/cors.go:29 +0xb0e
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1()
/go/src/github.com/hashicorp/vault/http/util.go:97 +0xf28
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1()
/go/src/github.com/hashicorp/vault/http/util_ent.go:81 +0x7e3
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
github.com/hashicorp/vault/http.wrapGenericHandler.func1()
/go/src/github.com/hashicorp/vault/http/handler.go:465 +0x1843
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1()
/go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0xc1
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:2046 +0x4d
net/http.serverHandler.ServeHTTP()
/usr/local/go/src/net/http/server.go:2878 +0x89a
net/http.initALPNRequest.ServeHTTP()
/usr/local/go/src/net/http/server.go:3479 +0x34d
net/http.(*initALPNRequest).ServeHTTP()
<autogenerated>:1 +0x8f
net/http.Handler.ServeHTTP-fm()
/usr/local/go/src/net/http/server.go:87 +0x75
net/http.(*http2serverConn).runHandler()
/usr/local/go/src/net/http/h2_bundle.go:5832 +0xdd
net/http.(*http2serverConn).processHeaders·dwrap·31()
/usr/local/go/src/net/http/h2_bundle.go:5562 +0x64
ncabatoff
approved these changes
Nov 9, 2021
vishalnayak
approved these changes
Nov 9, 2021
stevendpclark
deleted the
stevendpclark/fix-data-race-alias-identity-store
branch
stevendpclark
added a commit
that referenced
this pull request
Nov 9, 2021
…as (#13093) - When loading an existing alias within processLocalAlias we aren't cloning the object from the memory store. There seems to be a data race within the function when calling entity.UpsertAlias and a concurrent invalidation routine. ================== WARNING: DATA RACE Read at 0x00c00bd03d08 by goroutine 94: google.golang.org/protobuf/internal/impl.pointer.Elem() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/pointer_unsafe.go:118 +0x2b3 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:76 +0x265 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:56 +0x12a google.golang.org/protobuf/internal/impl.(*MessageInfo).size() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:40 +0x95 google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:33 +0x6c google.golang.org/protobuf/proto.MarshalOptions.marshal() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:153 +0x1f3 google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:122 +0xa5 github.com/golang/protobuf/proto.marshalAppend() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xe4 github.com/golang/protobuf/proto.Marshal() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23 +0x64 github.com/hashicorp/vault/helper/identity.(*Entity).Clone() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:34 +0x150 github.com/hashicorp/vault/vault.(*IdentityStore).MemDBEntitiesByBucketKeyInTxn() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:1214 +0x306 github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate() /go/src/github.com/hashicorp/vault/vault/identity_store.go:216 +0xd6c github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate-fm() /go/src/github.com/hashicorp/vault/vault/identity_store.go:160 +0x6d github.com/hashicorp/vault/sdk/framework.(*Backend).InvalidateKey() /go/src/github.com/hashicorp/vault/sdk/framework/backend.go:347 +0x8a github.com/hashicorp/vault/vault.(*IdentityStore).InvalidateKey() <autogenerated>:1 +0x7d github.com/hashicorp/vault/vault.(*Core).asyncInvalidateKey() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:58 +0x390 github.com/hashicorp/vault/vault.(*Core).asyncInvalidateHandler() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:71 +0x9b github.com/hashicorp/vault/vault.startReplicationEnt·dwrap·453() /go/src/github.com/hashicorp/vault/vault/replication_util_ent.go:331 +0x71 Previous write at 0x00c00bd03d08 by goroutine 52: github.com/hashicorp/vault/helper/identity.(*Entity).UpsertAlias() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:55 +0x271 github.com/hashicorp/vault/vault.(*IdentityStore).processLocalAlias() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:720 +0x672 github.com/hashicorp/vault/vault.possiblyForwardEntityCreation() /go/src/github.com/hashicorp/vault/vault/request_handling_util_ent.go:230 +0x286 github.com/hashicorp/vault/vault.(*Core).handleLoginRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:1345 +0x234a github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:607 +0x1a11 github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:442 +0x5b5 github.com/hashicorp/vault/vault.(*Core).HandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:408 +0xf2 github.com/hashicorp/vault/http.request() /go/src/github.com/hashicorp/vault/http/handler.go:953 +0xb1 github.com/hashicorp/vault/http.handleLogicalInternal.func1() /go/src/github.com/hashicorp/vault/http/logical.go:341 +0xca net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.handleRequestForwarding.func1() /go/src/github.com/hashicorp/vault/http/handler.go:887 +0x4eb net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.(*ServeMux).ServeHTTP() /usr/local/go/src/net/http/server.go:2424 +0xc5 github.com/hashicorp/vault/http.wrapHelpHandler.func1() /go/src/github.com/hashicorp/vault/http/help.go:23 +0x281 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapCORSHandler.func1() /go/src/github.com/hashicorp/vault/http/cors.go:29 +0xb0e net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1() /go/src/github.com/hashicorp/vault/http/util.go:97 +0xf28 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1() /go/src/github.com/hashicorp/vault/http/util_ent.go:81 +0x7e3 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapGenericHandler.func1() /go/src/github.com/hashicorp/vault/http/handler.go:465 +0x1843 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1() /go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0xc1 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.serverHandler.ServeHTTP() /usr/local/go/src/net/http/server.go:2878 +0x89a net/http.initALPNRequest.ServeHTTP() /usr/local/go/src/net/http/server.go:3479 +0x34d net/http.(*initALPNRequest).ServeHTTP() <autogenerated>:1 +0x8f net/http.Handler.ServeHTTP-fm() /usr/local/go/src/net/http/server.go:87 +0x75 net/http.(*http2serverConn).runHandler() /usr/local/go/src/net/http/h2_bundle.go:5832 +0xdd net/http.(*http2serverConn).processHeaders·dwrap·31() /usr/local/go/src/net/http/h2_bundle.go:5562 +0x64
stevendpclark
added a commit
that referenced
this pull request
Nov 9, 2021
stevendpclark
added a commit
that referenced
this pull request
Nov 9, 2021
stevendpclark
added a commit
that referenced
this pull request
Nov 9, 2021
|
Changelog was added within #13095 |
stevendpclark
added a commit
that referenced
this pull request
Nov 9, 2021
…rocessLocalAlias (#13093) (#13094) * Address a data race issue within identity_store_util::processLocalAlias (#13093) - When loading an existing alias within processLocalAlias we aren't cloning the object from the memory store. There seems to be a data race within the function when calling entity.UpsertAlias and a concurrent invalidation routine. ================== WARNING: DATA RACE Read at 0x00c00bd03d08 by goroutine 94: google.golang.org/protobuf/internal/impl.pointer.Elem() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/pointer_unsafe.go:118 +0x2b3 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:76 +0x265 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:56 +0x12a google.golang.org/protobuf/internal/impl.(*MessageInfo).size() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:40 +0x95 google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:33 +0x6c google.golang.org/protobuf/proto.MarshalOptions.marshal() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:153 +0x1f3 google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:122 +0xa5 github.com/golang/protobuf/proto.marshalAppend() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xe4 github.com/golang/protobuf/proto.Marshal() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23 +0x64 github.com/hashicorp/vault/helper/identity.(*Entity).Clone() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:34 +0x150 github.com/hashicorp/vault/vault.(*IdentityStore).MemDBEntitiesByBucketKeyInTxn() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:1214 +0x306 github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate() /go/src/github.com/hashicorp/vault/vault/identity_store.go:216 +0xd6c github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate-fm() /go/src/github.com/hashicorp/vault/vault/identity_store.go:160 +0x6d github.com/hashicorp/vault/sdk/framework.(*Backend).InvalidateKey() /go/src/github.com/hashicorp/vault/sdk/framework/backend.go:347 +0x8a github.com/hashicorp/vault/vault.(*IdentityStore).InvalidateKey() <autogenerated>:1 +0x7d github.com/hashicorp/vault/vault.(*Core).asyncInvalidateKey() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:58 +0x390 github.com/hashicorp/vault/vault.(*Core).asyncInvalidateHandler() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:71 +0x9b github.com/hashicorp/vault/vault.startReplicationEnt·dwrap·453() /go/src/github.com/hashicorp/vault/vault/replication_util_ent.go:331 +0x71 Previous write at 0x00c00bd03d08 by goroutine 52: github.com/hashicorp/vault/helper/identity.(*Entity).UpsertAlias() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:55 +0x271 github.com/hashicorp/vault/vault.(*IdentityStore).processLocalAlias() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:720 +0x672 github.com/hashicorp/vault/vault.possiblyForwardEntityCreation() /go/src/github.com/hashicorp/vault/vault/request_handling_util_ent.go:230 +0x286 github.com/hashicorp/vault/vault.(*Core).handleLoginRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:1345 +0x234a github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:607 +0x1a11 github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:442 +0x5b5 github.com/hashicorp/vault/vault.(*Core).HandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:408 +0xf2 github.com/hashicorp/vault/http.request() /go/src/github.com/hashicorp/vault/http/handler.go:953 +0xb1 github.com/hashicorp/vault/http.handleLogicalInternal.func1() /go/src/github.com/hashicorp/vault/http/logical.go:341 +0xca net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.handleRequestForwarding.func1() /go/src/github.com/hashicorp/vault/http/handler.go:887 +0x4eb net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.(*ServeMux).ServeHTTP() /usr/local/go/src/net/http/server.go:2424 +0xc5 github.com/hashicorp/vault/http.wrapHelpHandler.func1() /go/src/github.com/hashicorp/vault/http/help.go:23 +0x281 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapCORSHandler.func1() /go/src/github.com/hashicorp/vault/http/cors.go:29 +0xb0e net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1() /go/src/github.com/hashicorp/vault/http/util.go:97 +0xf28 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1() /go/src/github.com/hashicorp/vault/http/util_ent.go:81 +0x7e3 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapGenericHandler.func1() /go/src/github.com/hashicorp/vault/http/handler.go:465 +0x1843 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1() /go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0xc1 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.serverHandler.ServeHTTP() /usr/local/go/src/net/http/server.go:2878 +0x89a net/http.initALPNRequest.ServeHTTP() /usr/local/go/src/net/http/server.go:3479 +0x34d net/http.(*initALPNRequest).ServeHTTP() <autogenerated>:1 +0x8f net/http.Handler.ServeHTTP-fm() /usr/local/go/src/net/http/server.go:87 +0x75 net/http.(*http2serverConn).runHandler() /usr/local/go/src/net/http/h2_bundle.go:5832 +0xdd net/http.(*http2serverConn).processHeaders·dwrap·31() /usr/local/go/src/net/http/h2_bundle.go:5562 +0x64 * Add missing changelog for pr #13093 (#13095)
stevendpclark
added a commit
that referenced
this pull request
Dec 20, 2021
- Testcase TestIdentityStore_LocalAliasInvalidations identified a data race issue. - This reverts the previous attempt to address the issue from #13093
mlevitt
pushed a commit
to mlevitt/vault
that referenced
this pull request
Dec 23, 2021
…rocessLocalAlias (hashicorp#13093) (hashicorp#13094) * Address a data race issue within identity_store_util::processLocalAlias (hashicorp#13093) - When loading an existing alias within processLocalAlias we aren't cloning the object from the memory store. There seems to be a data race within the function when calling entity.UpsertAlias and a concurrent invalidation routine. ================== WARNING: DATA RACE Read at 0x00c00bd03d08 by goroutine 94: google.golang.org/protobuf/internal/impl.pointer.Elem() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/pointer_unsafe.go:118 +0x2b3 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:76 +0x265 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:56 +0x12a google.golang.org/protobuf/internal/impl.(*MessageInfo).size() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:40 +0x95 google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:33 +0x6c google.golang.org/protobuf/proto.MarshalOptions.marshal() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:153 +0x1f3 google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:122 +0xa5 github.com/golang/protobuf/proto.marshalAppend() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xe4 github.com/golang/protobuf/proto.Marshal() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23 +0x64 github.com/hashicorp/vault/helper/identity.(*Entity).Clone() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:34 +0x150 github.com/hashicorp/vault/vault.(*IdentityStore).MemDBEntitiesByBucketKeyInTxn() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:1214 +0x306 github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate() /go/src/github.com/hashicorp/vault/vault/identity_store.go:216 +0xd6c github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate-fm() /go/src/github.com/hashicorp/vault/vault/identity_store.go:160 +0x6d github.com/hashicorp/vault/sdk/framework.(*Backend).InvalidateKey() /go/src/github.com/hashicorp/vault/sdk/framework/backend.go:347 +0x8a github.com/hashicorp/vault/vault.(*IdentityStore).InvalidateKey() <autogenerated>:1 +0x7d github.com/hashicorp/vault/vault.(*Core).asyncInvalidateKey() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:58 +0x390 github.com/hashicorp/vault/vault.(*Core).asyncInvalidateHandler() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:71 +0x9b github.com/hashicorp/vault/vault.startReplicationEnt·dwrap·453() /go/src/github.com/hashicorp/vault/vault/replication_util_ent.go:331 +0x71 Previous write at 0x00c00bd03d08 by goroutine 52: github.com/hashicorp/vault/helper/identity.(*Entity).UpsertAlias() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:55 +0x271 github.com/hashicorp/vault/vault.(*IdentityStore).processLocalAlias() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:720 +0x672 github.com/hashicorp/vault/vault.possiblyForwardEntityCreation() /go/src/github.com/hashicorp/vault/vault/request_handling_util_ent.go:230 +0x286 github.com/hashicorp/vault/vault.(*Core).handleLoginRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:1345 +0x234a github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:607 +0x1a11 github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:442 +0x5b5 github.com/hashicorp/vault/vault.(*Core).HandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:408 +0xf2 github.com/hashicorp/vault/http.request() /go/src/github.com/hashicorp/vault/http/handler.go:953 +0xb1 github.com/hashicorp/vault/http.handleLogicalInternal.func1() /go/src/github.com/hashicorp/vault/http/logical.go:341 +0xca net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.handleRequestForwarding.func1() /go/src/github.com/hashicorp/vault/http/handler.go:887 +0x4eb net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.(*ServeMux).ServeHTTP() /usr/local/go/src/net/http/server.go:2424 +0xc5 github.com/hashicorp/vault/http.wrapHelpHandler.func1() /go/src/github.com/hashicorp/vault/http/help.go:23 +0x281 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapCORSHandler.func1() /go/src/github.com/hashicorp/vault/http/cors.go:29 +0xb0e net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1() /go/src/github.com/hashicorp/vault/http/util.go:97 +0xf28 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1() /go/src/github.com/hashicorp/vault/http/util_ent.go:81 +0x7e3 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapGenericHandler.func1() /go/src/github.com/hashicorp/vault/http/handler.go:465 +0x1843 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1() /go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0xc1 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.serverHandler.ServeHTTP() /usr/local/go/src/net/http/server.go:2878 +0x89a net/http.initALPNRequest.ServeHTTP() /usr/local/go/src/net/http/server.go:3479 +0x34d net/http.(*initALPNRequest).ServeHTTP() <autogenerated>:1 +0x8f net/http.Handler.ServeHTTP-fm() /usr/local/go/src/net/http/server.go:87 +0x75 net/http.(*http2serverConn).runHandler() /usr/local/go/src/net/http/h2_bundle.go:5832 +0xdd net/http.(*http2serverConn).processHeaders·dwrap·31() /usr/local/go/src/net/http/h2_bundle.go:5562 +0x64 * Add missing changelog for pr hashicorp#13093 (hashicorp#13095)
heppu
pushed a commit
to heppu/vault
that referenced
this pull request
Jan 13, 2022
…ashicorp#13476) * Attempt to address a data race issue within identity store * Testcase TestIdentityStore_LocalAliasInvalidations identified a data race issue. * This reverts the previous attempt to address the issue from hashicorp#13093
qk4l
pushed a commit
to qk4l/vault
that referenced
this pull request
Feb 4, 2022
…as (hashicorp#13093) - When loading an existing alias within processLocalAlias we aren't cloning the object from the memory store. There seems to be a data race within the function when calling entity.UpsertAlias and a concurrent invalidation routine. ================== WARNING: DATA RACE Read at 0x00c00bd03d08 by goroutine 94: google.golang.org/protobuf/internal/impl.pointer.Elem() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/pointer_unsafe.go:118 +0x2b3 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:76 +0x265 google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:56 +0x12a google.golang.org/protobuf/internal/impl.(*MessageInfo).size() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:40 +0x95 google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:33 +0x6c google.golang.org/protobuf/proto.MarshalOptions.marshal() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:153 +0x1f3 google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend() /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:122 +0xa5 github.com/golang/protobuf/proto.marshalAppend() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xe4 github.com/golang/protobuf/proto.Marshal() /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23 +0x64 github.com/hashicorp/vault/helper/identity.(*Entity).Clone() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:34 +0x150 github.com/hashicorp/vault/vault.(*IdentityStore).MemDBEntitiesByBucketKeyInTxn() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:1214 +0x306 github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate() /go/src/github.com/hashicorp/vault/vault/identity_store.go:216 +0xd6c github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate-fm() /go/src/github.com/hashicorp/vault/vault/identity_store.go:160 +0x6d github.com/hashicorp/vault/sdk/framework.(*Backend).InvalidateKey() /go/src/github.com/hashicorp/vault/sdk/framework/backend.go:347 +0x8a github.com/hashicorp/vault/vault.(*IdentityStore).InvalidateKey() <autogenerated>:1 +0x7d github.com/hashicorp/vault/vault.(*Core).asyncInvalidateKey() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:58 +0x390 github.com/hashicorp/vault/vault.(*Core).asyncInvalidateHandler() /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:71 +0x9b github.com/hashicorp/vault/vault.startReplicationEnt·dwrap·453() /go/src/github.com/hashicorp/vault/vault/replication_util_ent.go:331 +0x71 Previous write at 0x00c00bd03d08 by goroutine 52: github.com/hashicorp/vault/helper/identity.(*Entity).UpsertAlias() /go/src/github.com/hashicorp/vault/helper/identity/identity.go:55 +0x271 github.com/hashicorp/vault/vault.(*IdentityStore).processLocalAlias() /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:720 +0x672 github.com/hashicorp/vault/vault.possiblyForwardEntityCreation() /go/src/github.com/hashicorp/vault/vault/request_handling_util_ent.go:230 +0x286 github.com/hashicorp/vault/vault.(*Core).handleLoginRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:1345 +0x234a github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:607 +0x1a11 github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:442 +0x5b5 github.com/hashicorp/vault/vault.(*Core).HandleRequest() /go/src/github.com/hashicorp/vault/vault/request_handling.go:408 +0xf2 github.com/hashicorp/vault/http.request() /go/src/github.com/hashicorp/vault/http/handler.go:953 +0xb1 github.com/hashicorp/vault/http.handleLogicalInternal.func1() /go/src/github.com/hashicorp/vault/http/logical.go:341 +0xca net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.handleRequestForwarding.func1() /go/src/github.com/hashicorp/vault/http/handler.go:887 +0x4eb net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.(*ServeMux).ServeHTTP() /usr/local/go/src/net/http/server.go:2424 +0xc5 github.com/hashicorp/vault/http.wrapHelpHandler.func1() /go/src/github.com/hashicorp/vault/http/help.go:23 +0x281 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapCORSHandler.func1() /go/src/github.com/hashicorp/vault/http/cors.go:29 +0xb0e net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1() /go/src/github.com/hashicorp/vault/http/util.go:97 +0xf28 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1() /go/src/github.com/hashicorp/vault/http/util_ent.go:81 +0x7e3 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/vault/http.wrapGenericHandler.func1() /go/src/github.com/hashicorp/vault/http/handler.go:465 +0x1843 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1() /go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0xc1 net/http.HandlerFunc.ServeHTTP() /usr/local/go/src/net/http/server.go:2046 +0x4d net/http.serverHandler.ServeHTTP() /usr/local/go/src/net/http/server.go:2878 +0x89a net/http.initALPNRequest.ServeHTTP() /usr/local/go/src/net/http/server.go:3479 +0x34d net/http.(*initALPNRequest).ServeHTTP() <autogenerated>:1 +0x8f net/http.Handler.ServeHTTP-fm() /usr/local/go/src/net/http/server.go:87 +0x75 net/http.(*http2serverConn).runHandler() /usr/local/go/src/net/http/h2_bundle.go:5832 +0xdd net/http.(*http2serverConn).processHeaders·dwrap·31() /usr/local/go/src/net/http/h2_bundle.go:5562 +0x64
qk4l
pushed a commit
to qk4l/vault
that referenced
this pull request
Feb 4, 2022
qk4l
pushed a commit
to qk4l/vault
that referenced
this pull request
Feb 4, 2022
…ashicorp#13476) * Attempt to address a data race issue within identity store * Testcase TestIdentityStore_LocalAliasInvalidations identified a data race issue. * This reverts the previous attempt to address the issue from hashicorp#13093
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cloning the object from the memory store. There seems to be a data
race within the function when calling entity.UpsertAlias and
a concurrent invalidation routine.