Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Image pull fails when repository answers with optional unsupported WWW-Authenticate header #4187

Open
kosi2801 opened this issue Feb 16, 2024 · 0 comments · May be fixed by #4188
Open

Image pull fails when repository answers with optional unsupported WWW-Authenticate header #4187

kosi2801 opened this issue Feb 16, 2024 · 0 comments · May be fixed by #4188
Labels
priority: p2 type:bug

Comments

@kosi2801
Copy link

kosi2801 commented Feb 16, 2024 •
edited

Environment:

  • Jib version: 3.4.0
  • Build tool: Maven 3.6.3
  • OS: WSL2, Kali-Linux, 5.15.146.1-microsoft-standard-WSL2

Description of the issue:
Our Docker repository supports Basic Auth as authentication method, that has worked well for quite some time now. Recently because of network change reasons, the server also offers NTLM/Kerberos authentication.
Unfortunately the jib-maven-plugin now fails when trying to download an image with:
Failed to authenticate with registry xyz.registry/xyz-image because: 'Bearer' was not found in the 'WWW-Authenticate' header, tried to parse: Negotiate

Tracing the network communication shows following response from the Docker server that clearly contains a Basic Auth possibility:

...
Feb 15, 2024 5:06:33 PM com.google.api.client.http.HttpRequest execute
KONFIGURATION: -------------- REQUEST  --------------
GET https://xyz.registry/xyz-image
Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip
User-Agent: jib 3.4.0 jib-maven-plugin Google-HTTP-Java-Client/1.42.2 (gzip)

Feb 15, 2024 5:06:33 PM com.google.api.client.http.HttpRequest execute
KONFIGURATION: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 3.4.0 jib-maven-plugin Google-HTTP-Java-Client/1.42.2 (gzip)' -- 'https://xyz.registry/xyz-image'
Feb 15, 2024 5:06:33 PM com.google.api.client.http.HttpResponse <init>
KONFIGURATION: -------------- RESPONSE --------------
HTTP/1.1 401 Unauthorized
Server: nginx
Date: Thu, 15 Feb 2024 16:06:32 GMT
Content-Type: text/html
Content-Length: 172
Connection: keep-alive
Keep-Alive: timeout=120
WWW-Authenticate: Negotiate
WWW-Authenticate: Basic realm="xyz.registry"

Feb 15, 2024 5:06:33 PM com.google.api.client.util.LoggingByteArrayOutputStream close
KONFIGURATION: Total: 172 bytes
Feb 15, 2024 5:06:33 PM com.google.api.client.util.LoggingByteArrayOutputStream close
KONFIGURATION: <html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
...

Changing back the configuration so that only a "Basic" authentication is contained in the response header makes everything work again properly.

The problem seems to be that all retrievals in the code for "WWW-Authenticate" header only fetches the first occurence, like ex.getHttpResponseException().getHeaders().getAuthenticate(); and ignore all other/later possibilities.
If that first authenticate option is not supported by jib, no further attempts to check the other options are done and the whole download fails.

Possibly relevant source locations:

  • com/google/cloud/tools/jib/registry/AuthenticationMethodRetriever.java:96
  • com/google/cloud/tools/jib/builder/steps/PullBaseImageStep.java:174
  • com/google/cloud/tools/jib/registry/RegistryClient.java:635

Expected behavior:

The repository authentication should ignore unsupported authentication methods (like "Negotiate") and try the other options if available.

Steps to reproduce:

  1. configure the repository so that it returns an unsupported WWW-Authenticate header as first line additional to an already existing WWW-Authenticate Basic Auth header (maybe with an nginx-proxy that inserts such headers artifically)
  2. build a project with jit-maven-plugin that downloads an image from that docker repository using Basic Auth

Log output: 

...
[DEBUG] TIMING	Pulling base image manifest
[INFO] Getting manifest for base image xyz.registry/xyz-image...
[DEBUG] TIMING	trying mirrors
[DEBUG] TIMED	trying mirrors : 1.0 ms
Feb 15, 2024 5:06:33 PM com.google.api.client.http.HttpRequest execute
KONFIGURATION: -------------- REQUEST  --------------
GET https://xyz.registry/xyz-image
Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip
User-Agent: jib 3.4.0 jib-maven-plugin Google-HTTP-Java-Client/1.42.2 (gzip)

Feb 15, 2024 5:06:33 PM com.google.api.client.http.HttpRequest execute
KONFIGURATION: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 3.4.0 jib-maven-plugin Google-HTTP-Java-Client/1.42.2 (gzip)' -- 'https://xyz.registry/xyz-image'
Feb 15, 2024 5:06:33 PM com.google.api.client.http.HttpResponse <init>
KONFIGURATION: -------------- RESPONSE --------------
HTTP/1.1 401 Unauthorized
Server: nginx
Date: Thu, 15 Feb 2024 16:06:32 GMT
Content-Type: text/html
Content-Length: 172
Connection: keep-alive
Keep-Alive: timeout=120
WWW-Authenticate: Negotiate
WWW-Authenticate: Basic realm="xyz.registry"

Feb 15, 2024 5:06:33 PM com.google.api.client.util.LoggingByteArrayOutputStream close
KONFIGURATION: Total: 172 bytes
Feb 15, 2024 5:06:33 PM com.google.api.client.util.LoggingByteArrayOutputStream close
KONFIGURATION: <html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>

[INFO] The base image requires auth. Trying again for xyz.registry/xyz-image...
[INFO] Using credentials from Maven settings file for xyz.registry/xyz-image
[DEBUG] WWW-Authenticate for xyz.registry/xyz-image: Negotiate
[DEBUG] TIMED	Pulling base image manifest : 934.0 ms
[DEBUG] TIMING	Preparing application layer builders
[DEBUG] TIMED	Preparing application layer builders : 1.0 ms
[INFO] Building dependencies layer...
[DEBUG] TIMING	Building dependencies layer
[DEBUG] TIMED	Building dependencies layer : 163.0 ms
[INFO] Building snapshot dependencies layer...
[DEBUG] TIMING	Building snapshot dependencies layer
[DEBUG] TIMED	Building snapshot dependencies layer : 4.0 ms
[INFO] Building project dependencies layer...
[DEBUG] TIMING	Building project dependencies layer
[DEBUG] Building project dependencies layer built sha256:xxxyyyzzz
[DEBUG] TIMED	Building project dependencies layer : 33.0 ms
[INFO] Building resources layer...
[DEBUG] TIMING	Building resources layer
[DEBUG] Building resources layer built sha256:xxxyyyzzz
[DEBUG] TIMED	Building resources layer : 62.0 ms
[INFO] Building classes layer...
[DEBUG] TIMING	Building classes layer
[DEBUG] TIMED	Building classes layer : 11.0 ms
[INFO] Building jvm arg files layer...
[DEBUG] TIMING	Building jvm arg files layer
[DEBUG] TIMED	Building jvm arg files layer : 0.0 ms
[INFO] Building extra files layer...
[DEBUG] TIMING	Building extra files layer
[DEBUG] TIMED	Building extra files layer : 0.0 ms
[DEBUG] TIMED	Building image to Docker daemon : 1298.0 ms
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for xyz master-SNAPSHOT:
[INFO] 
[INFO] xyz ................................................ SUCCESS [  1.796 s]
<...>
[INFO] xyz-dev ............................................ SUCCESS [  0.920 s]
[INFO] xyz-app ............................................ FAILURE [ 34.480 s]
[INFO] xyz-tests .......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  47.180 s
[INFO] Finished at: 2024-02-15T17:06:34+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:dockerBuild (build-dev-image) on project xyz-app: Build to Docker daemon failed: Failed to authenticate with registry xyz.registry/xyz-image because: 'Bearer' was not found in the 'WWW-Authenticate' header, tried to parse: Negotiate -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:dockerBuild (build-dev-image) on project xyz-app: Build to Docker daemon failed
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: Build to Docker daemon failed
    at com.google.cloud.tools.jib.maven.BuildDockerMojo.execute (BuildDockerMojo.java:167)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: com.google.cloud.tools.jib.api.RegistryAuthenticationFailedException: Failed to authenticate with registry xyz.registry/xyz-image because: 'Bearer' was not found in the 'WWW-Authenticate' header, tried to parse: Negotiate
    at com.google.cloud.tools.jib.registry.RegistryAuthenticator.newRegistryAuthenticationFailedException (RegistryAuthenticator.java:109)
    at com.google.cloud.tools.jib.registry.RegistryAuthenticator.fromAuthenticationMethod (RegistryAuthenticator.java:85)
    at com.google.cloud.tools.jib.registry.RegistryClient.authPullByWwwAuthenticate (RegistryClient.java:390)
    at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call (PullBaseImageStep.java:178)
    at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call (PullBaseImageStep.java:70)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:131)
    at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:75)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:82)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute (MoreExecutors.java:327)
    at java.util.concurrent.AbstractExecutorService.submit (AbstractExecutorService.java:134)
    at com.google.common.util.concurrent.AbstractListeningExecutorService.submit (AbstractListeningExecutorService.java:79)
    at com.google.common.util.concurrent.AbstractListeningExecutorService.submit (AbstractListeningExecutorService.java:37)
    at com.google.cloud.tools.jib.builder.steps.StepsRunner.pullBaseImages (StepsRunner.java:302)
    at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$run$2 (StepsRunner.java:218)
    at java.util.ArrayList.forEach (ArrayList.java:1259)
    at com.google.cloud.tools.jib.builder.steps.StepsRunner.run (StepsRunner.java:218)
    at com.google.cloud.tools.jib.api.Containerizer.run (Containerizer.java:406)
    at com.google.cloud.tools.jib.api.JibContainerBuilder.containerize (JibContainerBuilder.java:610)
    at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild (JibBuildRunner.java:229)
    at com.google.cloud.tools.jib.maven.BuildDockerMojo.execute (BuildDockerMojo.java:103)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :xyz-app

Additional Information:

I have prepared a local fix changing above mentioned locations in a way, that if the desired action cannot be performed it is re-attempted with the next option in the WWW-Authenticate headers list.
A pull request will be posted shortly.
kosi2801 pushed a commit to kosi2801/jib that referenced this issue Feb 16, 2024
fix: primitive support for multiple WWW-Authenticate response heade…
1feb4b1 
…rs (GoogleContainerTools#4187)

* fix: try all authenticate methods from the `WWW-Authenticate` response header until one succeeds
kosi2801 added a commit to kosi2801/jib that referenced this issue Feb 16, 2024
@kosi2801
fix: primitive support for multiple WWW-Authenticate response heade…
b7eb512 
…rs (GoogleContainerTools#4187)

* fix: try all authenticate methods from the `WWW-Authenticate` response header until one succeeds
@kosi2801 kosi2801 linked a pull request Feb 16, 2024 that will close this issue
Open
4 tasks
kosi2801 added a commit to kosi2801/jib that referenced this issue Feb 28, 2024
@kosi2801
fix: primitive support for multiple WWW-Authenticate response heade…
01bc3ab 
…rs (GoogleContainerTools#4187)

* fix: try all authenticate methods from the `WWW-Authenticate` response header until one succeeds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority: p2 type:bug
Projects
None yet
Milestone
No milestone
2 participants
@kosi2801 @lqiu96