Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Identity: prepublish jwt signing keys #12414

Merged
merged 15 commits into from Sep 9, 2021

Conversation

fairclothjm
Copy link
Contributor

@fairclothjm fairclothjm commented Aug 24, 2021

Addresses #11438 which reported that OIDC key rotations happen up to 60 seconds after they are scheduled. This becomes an issue because the Cache-Control header returned on calls to identity/oidc/.well-known/keys will return Cache-Control: no-store for the 0 to 60 seconds between the scheduled rotation time and when rotation happens. This causes a thundering herd of server requests to update their JWKS keys.

This solution updates the Cache-Control logic to return a random value between 0 and the lowest rotation_period and validation_ttl of all keys on the vault cluster, to smooth out the rate servers call Vault for JWKS keys and avoid the thundering herd issues.

@fairclothjm fairclothjm requested review from vinay-gopalan, calvn, austingebauer and a team and removed request for a team August 24, 2021 14:45
vault/identity_store_oidc.go Outdated Show resolved Hide resolved
vault/identity_store_oidc.go Outdated Show resolved Hide resolved
key.KeyRing = append(key.KeyRing, &expireableKey{KeyID: signingKey.Public().KeyID})
key.KeyRing = append(key.KeyRing, &expireableKey{
KeyID: signingKey.Public().KeyID,
ExpireAt: now.Add(key.RotationPeriod).Add(key.VerificationTTL),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this means that we're calculating the ExpireAt by factoring the rotation period early on instead of just appending the VerificationTTL upon rotation like it currently is. What happens if there's an update on the named key's rotation_period after the key has been created (i.e. by an update)?

vault/identity_store_oidc.go Outdated Show resolved Hide resolved
Copy link
Member

@austingebauer austingebauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few comments to get started. Still looking through this and wrapping my head around the approach. Will give another pass shortly. Ty!

vault/identity_store_oidc.go Outdated Show resolved Hide resolved
vault/identity_store_oidc.go Outdated Show resolved Hide resolved
vault/identity_store_oidc.go Outdated Show resolved Hide resolved
vault/identity_store_oidc.go Outdated Show resolved Hide resolved
@vercel vercel bot temporarily deployed to Preview – vault September 1, 2021 19:28 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook September 1, 2021 19:28 Inactive
@vercel vercel bot temporarily deployed to Preview – vault September 1, 2021 19:32 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook September 1, 2021 19:32 Inactive
@vercel vercel bot temporarily deployed to Preview – vault September 1, 2021 21:13 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook September 1, 2021 21:13 Inactive
@vercel vercel bot temporarily deployed to Preview – vault September 2, 2021 14:37 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook September 2, 2021 14:37 Inactive
@vercel vercel bot temporarily deployed to Preview – vault September 3, 2021 15:41 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook September 3, 2021 15:41 Inactive
@vercel vercel bot temporarily deployed to Preview – vault September 7, 2021 21:05 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook September 7, 2021 21:05 Inactive
@fairclothjm fairclothjm marked this pull request as ready for review September 8, 2021 15:46
Copy link
Member

@austingebauer austingebauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@@ -0,0 +1,3 @@
```release-note:bug
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I wonder if this is an enhancement instead of a bug? I think the implementation before functioned as designed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with that

@vercel vercel bot temporarily deployed to Preview – vault September 9, 2021 13:54 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook September 9, 2021 13:54 Inactive
@fairclothjm fairclothjm merged commit 7b49d57 into main Sep 9, 2021
@fairclothjm fairclothjm deleted the identity-prepublish_jwt_signing_keys branch September 9, 2021 18:47
jartek pushed a commit to jartek/vault that referenced this pull request Sep 11, 2021
* pre-publish new signing keys for `rotation_period` of time before using

* Work In Progress: Prepublish JWKS and even cache control

* remove comments

* use math/rand instead of math/big

* update tests

* remove debug comment

* refactor cache control logic into func

* don't set expiry when create/update key

* update cachecontrol name in oidccache for test

* fix bug in periodicfunc test case

* add changelog

* remove confusing comment

* add logging and comments

* update change log from bug to improvement

Co-authored-by: Ian Ferguson <ian.ferguson@datadoghq.com>
@ianferguson
Copy link
Contributor

thanks for picking this up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants