INFO[0101] failed to verify token: token signed by untrusted key with ID:

[zfLQ2qx2]

Description

I'm testing updating to a recent build of docker-registry but I'm getting stuck on the authorization part. The error in the file is:

INFO[0101] failed to verify token: token signed by untrusted key with ID: "xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx"

• I've generated a completely new self-signed certificates w/ new a RSA 2048 key
• I've verified the key and certificate modulus matches
• I've verified that both the registry and auth service have the same key and cert, with matching modulus
• I've verified that the auth service is returning a positive response with a JWT token signed by the right key

The auth service is using docker/libtrust to generate the keyid:
https://github.com/cesanta/docker_auth/blob/38e7252690dc31ab7ccd4185a2dead973099f37c/auth_server/server/server.go#L384
https://github.com/cesanta/docker_auth/blob/38e7252690dc31ab7ccd4185a2dead973099f37c/auth_server/server/config.go#L88

So it seems to be docker-registry itself making the determination that the key is untrusted. However it doesn't generate any errors or warnings when loading the key, only when processing the auth response. The error message doesn't provide any hints as to why it is untrusted, and there are no other relevant messages in the log. This last worked using a build of docker-registry from early 2022 (commit c202b9b), the only change is using a March 2024 build (commit 663b430) - though I also tried a newer certificate to see if it gave a different result.

Reproduce

Setting up a local registry can be complex, I'll wait to see if its something simple like docker-registry only supports rsa-4096, or has banned values for common name, etc.

Expected behavior

The server should accept the positive response from the auth server and allow access.

registry version

/usr/bin/docker-registry github.com/distribution/distribution/v3 v3.0.0-alpha.1.m+unknown

Additional Info

No response

[zfLQ2qx2]

Hmm.. it looks like a workaround might be to have the auth service not send a key id, or to have the registry not validate it, but is that a good solution?

token signed by untrusted key with ID

[milosgajdos]

I feel like your trusted keys are not being loaded properly or you're not loading them at all:

distribution/registry/auth/token/accesscontroller.go

Lines 282 to 287 in d9815da

	trustedKeys := make(map[string]crypto.PublicKey)
	if jwks != nil {
	for _, key := range jwks.Keys {
	trustedKeys[key.KeyID] = key.Public()
	}
	}

You need to provide a path to your JWKS and pass it to the config file via the jwks config:

distribution/registry/auth/token/accesscontroller.go

Lines 265 to 270 in d9815da

	if config.jwks != "" {
	jwks, err = getJwks(config.jwks)
	if err != nil {
	return nil, err
	}
	}

You can of course "just" rely on the Cert bundle path to which is also passed via a config param:

distribution/registry/auth/token/accesscontroller.go

Lines 258 to 263 in d9815da

	if config.rootCertBundle != "" {
	rootCerts, err = getRootCerts(config.rootCertBundle)
	if err != nil {
	return nil, err
	}
	}

[zfLQ2qx2]

@milosgajdos I tried that and if the key is not present in the registry config it results in a panic - "panic: unable to configure authorization (token): token auth requires at least one token signing key". Likewise on the auth server I can see that the KeyID changes if I update the key but remains stable if I don't, so I'm pretty confident it is loading the file. I tried not sending the keyID and that resulted in the message "INFO[0050] failed to verify token: go-jose/go-jose: no x5c header present in message".

I am wondering if the format of the KeyID field changed and docker-registry can no longer handle values produced by a different method.

[zfLQ2qx2]

This seems to be the breaking change: #4096

Docker-registry can no longer process the kid values that were produced by the docker-libtrust library. It's not clear if its an issue of making the authentication service produce kid values the registry does understand, adding some compatibility code to honor the format that docker-libtrust used (the presence of colons seems to to be an indicator that its a docker-libtrust style id, or if there are other changes required such as the x5c field referenced above.

[zfLQ2qx2]

Confirmed that if I run https://github.com/milosgajdos/distribution/tree/1d410148efe6d1b7fd56457507a9dd465b105ec4 instead, authentication works again.

[milosgajdos]

if the key is not present in the registry config it results in a panic - "panic: unable to configure authorization (token): token auth requires at least one token signing key"

This is the correct behaviour -- how do you want to verify the token if you don't provide the trusted keys to registry?

I am wondering if the format of the KeyID field changed and docker-registry can no longer handle values produced by a different method

FYI: we have deprecated libstrust because it's no longer active and in fact, it has been archived by Docker. But I digress.

The format of the keyID is irrelevant here as long as the id is unique within the JWKs keys field. In my original answer I've linked to the JWKs spec: https://datatracker.ietf.org/doc/html/rfc7517#section-5

auth0 have a nice breakdown of what the JWKs look like: https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-set-properties

We load the JWKs we find in the file specified via the config:

distribution/registry/auth/token/accesscontroller.go

Lines 282 to 287 in d9815da

	trustedKeys := make(map[string]crypto.PublicKey)
	if jwks != nil {
	for _, key := range jwks.Keys {
	trustedKeys[key.KeyID] = key.Public()
	}
	}

I tried not sending the keyID and that resulted in the message "INFO[0050] failed to verify token: go-jose/go-jose: no x5c header present in message".

If you don't send the keyID and load the JWKs then the verification should correctly fail - how is the registry to know what key to verify if it doesn't have the keyID to look it up in its trusted keys? The error you are seeing happens when you rely on TLS certs but again fail to get registry to load them so it can use them for verification.

[markuskirch]

Description

@milosgajdos we're encountering the same problem while testing 3.0.0-alpha.1:

We run an auth_server and a registry from the following compose configuration.

• The given configuration worked fine with the 2.8.3 image
• This error occurs after switching to 3.0.0-alpha.1, nothing else changed
• The auth_server returns a JWT as expected
• Registry authentication fails however: Error response from daemon: Get "http://<registry_url>:5000/v2/": unauthorized: Auth failed.

Logs

auth_server logs:

registry-auth-server-1  | I0507 07:59:47.605332       1 server.go:429] Request: &{Method:GET URL:/auth?account=admin&client_id=docker&offline_token=true&service=Docker+registry Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[gzip] Authorization:[Basic YWRtaW46YmFkbWlu] Connection:[close] User-Agent:[docker/26.1.1 go/go1.21.9 git-commit/ac2de55 kernel/6.5.0-28-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.1 \(linux\))]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:true Host:localhost:5001 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:172.23.0.1:44192 RequestURI:/auth?account=admin&client_id=docker&offline_token=true&service=Docker+registry TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc0002041e0}
registry-auth-server-1  | I0507 07:59:47.605455       1 server.go:483] Auth request: {admin:***@172.23.0.1:44192 []}
registry-auth-server-1  | I0507 07:59:47.606915       1 server.go:317] Authn static admin -> true, map[], <nil>
registry-auth-server-1  | I0507 07:59:47.613530       1 server.go:424] New token for {admin:***@172.23.0.1:44192 []} map[]: {"iss":"MyOrg","sub":"admin","aud":"Docker registry","exp":1715069687,"nbf":1715068777,"iat":1715068787,"jti":"662458978209793452","access":[]}
registry-auth-server-1  | I0507 07:59:47.613570       1 server.go:519] {"access_token":"<header>.<claims>.<sig>"},"token":"<header>.<claims>.<sig>"}

registry logs:

registry-1  | 172.23.0.1 - - [07/May/2024:07:59:47 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/26.1.1 go/go1.21.9 git-commit/ac2de55 kernel/6.5.0-28-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.1 \\(linux\\))"
registry-1  | time="2024-05-07T07:59:47.614222092Z" level=info msg="failed to verify token: token signed by untrusted key with ID: \"<kid>\""
registry-1  | time="2024-05-07T07:59:47.614245918Z" level=warning msg="error authorizing context: invalid token" go.version=go1.21.5 http.request.host="localhost:5000" http.request.id=7336c5e0-eb8a-4a44-8473-2b2a3de3a9ac http.request.method=GET http.request.remoteaddr="172.23.0.1:50584" http.request.uri=/v2/ http.request.useragent="docker/26.1.1 go/go1.21.9 git-commit/ac2de55 kernel/6.5.0-28-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.1 \\(linux\\))" instance.id=7c42c16e-347b-4ca2-8a5e-e06fef8ffadd service=registry version=3.0.0-alpha.1
registry-1  | 172.23.0.1 - - [07/May/2024:07:59:47 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/26.1.1 go/go1.21.9 git-commit/ac2de55 kernel/6.5.0-28-generic os/linux arch/amd64 UpstreamClient(Docker-Client/26.1.1 \\(linux\\))"
registry-1  | 2024/05/07 07:59:51 traces export: Post "https://localhost:4318/v1/traces": dial tcp [::1]:4318: connect: connection refused

Reproduce

To reproduce, run the following configuration locally:

File Structure

├── .
│   ├── auth-server-config.yml
│   ├── docker-compose.yml
│   ├── certificate.pem
│   ├── private.key

certificate.pem and private.key

Generate SSL keypair using:

openssl req -x509 -newkey rsa:4096 -keyout private.key -out certificate.pem -sha256 -days 3650 -nodes -subj "/O=MyOrg/CN=localhost"

docker-compose.yml

services:
  registry-auth-server:
    image: cesanta/docker_auth:latest
    volumes:
      - ./auth-server-config.yml:/config/config.yml:ro
      - ./certificate.pem:/cert/certificate.pem:ro
      - ./private.key:/cert/private.key:ro
    command: --v=3 --alsologtostderr /config/config.yml
    ports:
      - "5001:5001"
  registry:
    image: registry:3.0.0-alpha.1 #no error with registry:2
    volumes:
      - ./certificate.pem:/cert/certificate.pem:ro
    ports:
      - "5000:5000"
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: true
      REGISTRY_AUTH: token
      REGISTRY_AUTH_TOKEN_REALM: "http://localhost:5001/auth"
      REGISTRY_AUTH_TOKEN_SERVICE: "Docker registry"
      REGISTRY_AUTH_TOKEN_ISSUER: "MyOrg"
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: "/cert/certificate.pem"

auth-server-config.yml

server:
  addr: ":5001"
token:
  issuer: "MyOrg"
  expiration: 900
  certificate: "/cert/certificate.pem"
  key: "/cert/private.key"
users:
  "admin":
    password: "$2y$05$LO.vzwpWC5LZGqThvEfznu8qhb5SGqvBSWY1J3yZ4AxtMRZ3kN5jC"  # badmin
acl:
  - match: {account: "admin"}
    actions: ["*"]
    comment: "Admin has full access to everything."

Reproduce Error

curl -u "admin:badmin" "http://localhost:5001/auth"

{"access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkpOVzI6SlJWWjpBTFNZOlZNQzU6MkVHVTpCQU5MOlJVQUw6VlIzSTo2MlAzOlIyQ0w6R0FWUjpQTFRDIn0.eyJpc3MiOiJNeU9yZyIsInN1YiI6ImFkbWluIiwiYXVkIjoiIiwiZXhwIjoxNzE1MDc4OTk2LCJuYmYiOjE3MTUwNzgwODYsImlhdCI6MTcxNTA3ODA5NiwianRpIjoiNDY2MDEyNjgxMDYxMTUyMTEwNiIsImFjY2VzcyI6W119.JJ9OIiVpfHdMqp71Dz0xhrhmUCBxSeu_z5pl_Bk6ualVilGaXcC7NOJvNSDxqcLM-rcsdCTJKsdi5zYUg3F6uglZbudpzwvZ_2SUKVIDRIkTIfIhhPome_JWMUg1oB0cjx3Xz7ioDoHE_nDLprzSf2GbjqNKoadtLnIWG0PZWZe6F4qWsCk1713nTqIYN3nw31AbMUIrQEp0U5Z0YeDHnKQ50EviBKQq73IraBtlm0zvSdN2qXuyriniwjdnUrftyl1oaJRM2PnZpefh97j49Rh76RukiyjnXC6XsxWUoz1lf8kxky5bZD-uVYNkR-vzk4f8KNyeZqrNmpMieEje20Jc_JHFvT2zyg9NIaebPoz-fgvWEy1T1x7m7GRS8iB4cT5sQuex9fBuTmZYg2H0HZT-3toXmcaS9PRnuM7E2PrvIw8ZcE9GsqDacgyCXTQ4JGpSdNZOVLJ7LvCJJi5PyVWIJOCEkRHgYzA37_b1TajCdsKZcMMD0q9BjSq8sKKQRwkz_xcdZMtHe7nHGa_SjBP8OFHbF_04wKYhVM4ppe-VoPi4P4ppEj6jN_DhLpqUu3Zr1LfbywF8mgBECTNgugLlAMg6AyDh9xgWhUiwt1TbWlpxxEQj8p6z1bp_XZOMqFGEIfvUqgpxaIoreihfBkhzv7VcLf98hERYFVd3OAc","token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkpOVzI6SlJWWjpBTFNZOlZNQzU6MkVHVTpCQU5MOlJVQUw6VlIzSTo2MlAzOlIyQ0w6R0FWUjpQTFRDIn0.eyJpc3MiOiJNeU9yZyIsInN1YiI6ImFkbWluIiwiYXVkIjoiIiwiZXhwIjoxNzE1MDc4OTk2LCJuYmYiOjE3MTUwNzgwODYsImlhdCI6MTcxNTA3ODA5NiwianRpIjoiNDY2MDEyNjgxMDYxMTUyMTEwNiIsImFjY2VzcyI6W119.JJ9OIiVpfHdMqp71Dz0xhrhmUCBxSeu_z5pl_Bk6ualVilGaXcC7NOJvNSDxqcLM-rcsdCTJKsdi5zYUg3F6uglZbudpzwvZ_2SUKVIDRIkTIfIhhPome_JWMUg1oB0cjx3Xz7ioDoHE_nDLprzSf2GbjqNKoadtLnIWG0PZWZe6F4qWsCk1713nTqIYN3nw31AbMUIrQEp0U5Z0YeDHnKQ50EviBKQq73IraBtlm0zvSdN2qXuyriniwjdnUrftyl1oaJRM2PnZpefh97j49Rh76RukiyjnXC6XsxWUoz1lf8kxky5bZD-uVYNkR-vzk4f8KNyeZqrNmpMieEje20Jc_JHFvT2zyg9NIaebPoz-fgvWEy1T1x7m7GRS8iB4cT5sQuex9fBuTmZYg2H0HZT-3toXmcaS9PRnuM7E2PrvIw8ZcE9GsqDacgyCXTQ4JGpSdNZOVLJ7LvCJJi5PyVWIJOCEkRHgYzA37_b1TajCdsKZcMMD0q9BjSq8sKKQRwkz_xcdZMtHe7nHGa_SjBP8OFHbF_04wKYhVM4ppe-VoPi4P4ppEj6jN_DhLpqUu3Zr1LfbywF8mgBECTNgugLlAMg6AyDh9xgWhUiwt1TbWlpxxEQj8p6z1bp_XZOMqFGEIfvUqgpxaIoreihfBkhzv7VcLf98hERYFVd3OAc"}

docker login -u "admin" -p "badmin" localhost:5000

Error response from daemon: login attempt to http://localhost:5000/v2/ failed with status: 401 Unauthorized

=> Test this with the registry:2 image, and docker login with succeed. Nothing else changed.

JWT token (after joseBase64UrlDecode)

Format: <header>.<claims>.<sign>

header

{
    "typ": "JWT",
    "alg": "RS256",
    "kid": "JNW2:JRVZ:ALSY:VMC5:2EGU:BANL:RUAL:VR3I:62P3:R2CL:GAVR:PLTC"
}

claims

{
    "iss": "MyOrg",
    "sub": "admin",
    "aud": "",
    "exp": 1715078996,
    "nbf": 1715078086,
    "iat": 1715078096,
    "jti": "4660126810611521106",
    "access": []
}

sig

$\x9fN"%i|wL\xaa\x9e\xf5\x0f=1\x86\xb8fP qI\xeb\xbf\xcf\x9ae\xfc\x19:\xb9\xa9U\x8aQ\x9a]\xc0\xbb4\xe2o5 \xf1\xa9\xc2\xcc\xfa\xb7,t$\xc9*\xc7b\xe76\x14\x83qz\xba\tYn\xe7i\xcf\x0b\xd9\xffd\x94)R\x03D\x89\x13!\xf2!\x84\xfa&{\xf2V1H5\xa0\x1d\x1c\x8f\x1d\xd7\xcf\xb8\xa8\x0e\x81\xc4\xfep\xcb\xa6\xbc\xd2\x7fa\x9b\x8e\xa3J\xa1\xa7m.r\x16\x1bC\xd9Y\x97\xba\x17\x8a\x96\xb0)5\xef]\xe7N\xa2\x187y\xf0\xdfP\x1b1B+@JtS\x96ta\xe0\xc7\x9c\xa49\xd0K\xe2\x04\xa4*\xefr+h\x1be\x9bL\xefI\xd3v\xa9{\xb2\xae)\xe2\xc27gR\xb7\xed\xca]hh\x94L\xd8\xf9\xd9\xa5\xe7\xe1\xf7\xb8\xf8\xf5\x18{\xe9\x1b\xa4\x8b(\xe7\\.\x97\xb3\x15\x94\xa3=e\x7f\xc91\x93.[d?\xaeU\x83dG\xeb\xf3\x93\x87\xfc(\xdc\x9ef\xaa\xcd\x9a\x93"xH\xde\xdbB\\\xfc\x91\xc5\xbd=\xb3\xca\x0fM!\xa7\x9b>\x8c\xfe~\x0b\xd6\x13-S\xd7\x1e\xe6\xecdR\xf2 xq>lB\xe7\xb1\xf5\xf0nNfX\x83a\xf4\x1d\x94\xfe\xde\xda\x17\x99\xc6\x92\xf4\xf4g\xb8\xce\xc4\xd8\xfa\xef#\x0f\x19pOF\xb2\xa0\xdar\x0c\x82]48$jRt\xd6NT\xb2{.\xf0\x89&.O\xc9U\x88$\xe0\x84\x91\x11\xe0c07\xef\xf6\xf5M\xa8\xc2v\xc2\x99p\xc3\x03\xd2\xafA\x8d*\xbc\xb0\xa2\x90G\t3\xff\x17\x1dd\xcbG{\xb9\xc7\x19\xaf\xd2\x8c\x13\xfc8Q\xdb\x17\xfd8\xc0\xa6!T\xce)\xa5\xef\x95\xa0\xf8\xb8?\x8ai\x12>\xa37\xf0\xe1.\x9a\x94\xbbvk\xd4\xb7\xdb\xcb\x01|\x9a\x00D\t3`\xba\x02\xe5\x00\xc8:\x03 \xe1\xf7\x18\x16\x85H\xb0\xb7T\xdbZZq\xc4D#\xf2\x9e\xb3\xd5\xba\x7f]\x93\x8c\xa8Q\x84!\xfb\xd4\xaa\nqh\x8a+z(_\x06Hs\xbf\xb5\\-\xff|\x84DX\x15Ww8\x07

Can you reproduce the error?
Did the deprecation of libtrust lead to a change in the expected kid format intentionally?

Thanks for your help!

[milosgajdos]

Can you please format your message properly? It's impossible to parse useful info from it, I'm afraid.

Also, it's hard to investigate in general if I can't see the JWT your auth server is issuing.

[markuskirch]

@milosgajdos I'm very sorry for my cryptic comment. I just updated it with a detailed explanation of the error and a sample configuration to reproduce it locally.

Thanks for your great work maintaining this essential project!
Please let me know if there's anything I can do to help

[milosgajdos]

Thanks, that's useful. Off the top of my head, I can see that your JWT contains the optional kid value.

The "kid" (key ID) Header Parameter is a hint indicating which key
was used to secure the JWS. This parameter allows originators to
explicitly signal a change of key to recipients. The structure of
the "kid" value is unspecified. Its value MUST be a case-sensitive
string. Use of this Header Parameter is OPTIONAL.

When used with a JWK, the "kid" value is used to match a JWK "kid"
parameter value.

If you are going to include the kid, then you need to tell the registry that you trust the keys that sign that were used to sign it so we can verify it. That's what trusted keys config is for. You can read about JWKs here: https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-set-properties

Are you signing these keys with TLS certs?

Now I see where all these woes are coming from...The original code was checking the certs chain first and if that passed the validation, we'd just return; the current code checks the JWTs first and certs later, which surfaced a lot of issues to folks.

I've opened a PR in go-jose/go-jose that exposes the missing x5c header in JWT that would help us revert to the original logic, but beware of what you're trying to do in your code now; if you want to make things work as is now, you need to provide trusted keys to registry so registry can verify the kid in your JWT.

[markuskirch]

Thanks for your quick help!

I noticed earlier today that kid is an optional parameter in the JWS spec. Interesting.
The third-party token service we currently use is becoming less and less maintained, which doesn't make me feel too optimistic about opening a PR there. Seems like I may have to sit down and write my own after all...

Please excuse my foolish question: What exactly is the "trusted keys" config and where can I specify the registry to trust a certain key?
I found nothing promising around the auth section in the list of configuration examples.

We indeed use TLS certificates to generate tokens. In production, the registry is deployed behind a load-balancer that handles TLS for all services. Therefore, no http.tls.[] parameters are set in our registry configuration file.

[milosgajdos]

Yeah we're missing jwks section in the docs:

distribution/registry/auth/token/accesscontroller.go

Line 184 in c49220d

keys := []string{"realm", "issuer", "service", "rootcertbundle", "jwks"}

Basically, it lets you specify the path to the keys you can use for verifying the JWT tokens -- in the similar way as the cert bundle path is specified.

If you are using TLS certs to sign your tokens, then once my PR is merged into go-jose I'll open a PR to this repo and we'll switch the order of verification; shoehorning it without proper error handling would just add more tech debt

[markuskirch]

Thanks a lot!
I'm writing a script to convert my TLS key to the JWKS format and will test setting the jwks parameter in the registry config once this is done.

I also just opened #4345 with a minimal proposal to include the new parameter in the docs. I can also write a more detailed description if desired.