Root signing key rotation and prepublishing

[schmichael]

Nomad 1.7 uses a root encryption key to encrypt Variables at rest and a root signing key to sign Workload Identities.

These root keys should be rotated automatically using the following logic:

1. New root key should be generated at root_key_rotation_threshold / 2 and the public signing key published before use in the JWKS endpoint.
2. At root_key_rotation_threshold the prepublished key will be made active and the old active key will be made inactive
3. After root_key_rotation_threshold + root_key_gc_threshold after the old key was marked inactive, it should be garbage collected.
   • Update root_key_* docs to reflect that keys are not gc'd until rotation_threshold + gc_threshold are reached to avoid invalidating otherwise valid JWTs in use.
4. Jobspecs with an identity.tll > root_key_rotation_threshold should receive a Warning on submit.

Prior Art

hashicorp/vault#12414

[tgross]

What you're envisioning here should cover #19367 and #19368. I'm going to unassign myself from those and if you want, we can either close them out now or you can close them out with this issue.

[schmichael]

Thanks for linking things together Tim.

we can either close them out now or you can close them out with this issue.

I'm going to leave them open until this ships to ensure everything is buttoned up appropriately.