Docker CI jobs fail with "failed to fetch oauth token: ... 401 Unauthorized"

[teor2345]

Motivation

Some Docker CI jobs fail with:

error: failed to solve: failed to fetch oauth token: unexpected status: 401 Unauthorized
Error: buildx failed with: error: failed to solve: failed to fetch oauth token: unexpected status: 401 Unauthorized

The failures don't seem to have a consistent pattern.

Potential Fixes

TODO: work out how to fix this

[gustavovalverde]

I'll have to research further as this error might be misleading docker/build-push-action#586

[gustavovalverde]

More insight: google-github-actions/auth#160

Might be an issue with GCP

[teor2345]

This is still happening on the main branch, even after the Konlet fix:
https://github.com/ZcashFoundation/zebra/runs/6267974240?check_suite_focus=true

[gustavovalverde]

This failed after merging https://github.com/ZcashFoundation/zebra/runs/6274654773?check_suite_focus=true#step:9:2170, so#4254 wasn't a solution

[gustavovalverde]

Adding more information, as changing to GitHub Registry didn't solve the problem:

• registry errors lack detail moby/buildkit#2019
• Cache manifest fails to write after buildkit v0.7.2 moby/buildkit#2068
• Support remote cache based on OCI Image Manifest layout moby/buildkit#2251

[teor2345]

Let me know if there is anything I can do to help with this.

[dougdonohoe]

FYI to other internet seekers: I stumbled across this issue because I'm seeing a similar issue on GCP Cloud Build pushing to GCP Artifact Registry when the build is longer than an hour. I filed docker/buildx#1205 with the details.

There is a workaround (build without --push, stop builder, build again). Full details with reproducible example can be found at dougdonohoe/build-timeout.

[gustavovalverde]

@dougdonohoe if it helps you, OIDC tokens expire after an hour...so you'll end up with an expired token when the push is happening. Our workaround was increasing this limit for OAuth tokens

Here's the specific change we applied to fix this https://github.com/ZcashFoundation/zebra/pull/4370/files#diff-5b40193e09025fd041de92d38c976d0b469e3ee85bef45a71d05d8b5078b1ee8R91

[dougdonohoe]

@dougdonohoe if it helps you, OIDC tokens expire after an hour...so you'll end up with an expired token when the push is happening. Our workaround was increasing this limit for OAuth tokens

Here's the specific change we applied to fix this https://github.com/ZcashFoundation/zebra/pull/4370/files#diff-5b40193e09025fd041de92d38c976d0b469e3ee85bef45a71d05d8b5078b1ee8R91

Thanks @gustavovalverde - I did see your fix. I haven't found a similar way to extend timeouts in GCP Cloud Build yet, but I'm looking for it and am asking Google for help too.

[guyLee687]

I believe you can also extend the timeout for cloud build using:

gcloud auth print-access-token --impersonate-service-account=<SERVICE_ACCOUNT> --lifetime=12h | docker login -u oauth2accesstoken --password-stdin https://us-docker.pkg.dev
docker-credential-gcr configure-docker --registries="us-docker.pkg.dev"

and having the cloud build service account impersonate a separate service account with extended credentials.