Image Building with custom buildpacks fails with permission denied on Arch Linux

[bala-striva]

When trying to configure buildpacks for the bootBuildImage target (from: #25561 (comment))

bootBuildImage {
	buildpacks = [
	        'gcr.io/paketo-buildpacks/eclipse-openj9:latest',
			'paketo-buildpacks/java'
	]
}

will result in an error

> Task :bootBuildImage
Building image 'demo:latest'

 > Pulling builder image 'docker.io/paketobuildpacks/builder:base' ..................................................
 > Pulled builder image 'paketobuildpacks/builder@sha256:148282212248cd49988c0cdddff19ee7091f02b03b2b4bce52e5d68640e7b0c5'
 > Pulling run image 'docker.io/paketobuildpacks/run:base-cnb' ..................................................
 > Pulled run image 'paketobuildpacks/run@sha256:956341a43edf4029b29114502f7deb185527aed63e2a5ab94d86cd33abb34e56'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9:latest' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9@sha256:9ec7ecf806716d1416b6f9030dea08a33cf1ed2d35dff5c2c4cab0d7f46cd4d1'
 > Executing lifecycle version v0.11.3
 > Using build cache volume 'pack-cache-877d4fc3db41.build'

 > Running creator
    [creator]     ===> DETECTING
    [creator]     ERROR: failed to lookup buildpack.toml for buildpack 'paketo-buildpacks/eclipse-openj9@8.0.0': open /cnb/buildpacks/paketo-buildpacks_eclipse-openj9/8.0.0/buildpack.toml: permission denied

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

Without custom buildpacks everything works fine and the image is created.

Checking the docker image gcr.io/paketo-buildpacks/eclipse-openj9:latest it seems that the buildpack.toml is in place with read all rights.

• Spring boot 2.5
• Gradle 7.0

[scottfrederick]

@bala-striva I can't reproduce the error you are getting. When I run bootBuildImage with those buildpacks it works fine, with the same SHAs for the builder, run, and buildpack images:

Building image 'docker.io/library/build-demo:0.0.1-SNAPSHOT'

 > Pulling builder image 'docker.io/paketobuildpacks/builder:base' ..................................................
 > Pulled builder image 'paketobuildpacks/builder@sha256:148282212248cd49988c0cdddff19ee7091f02b03b2b4bce52e5d68640e7b0c5'
 > Pulling run image 'docker.io/paketobuildpacks/run:base-cnb' ..................................................
 > Pulled run image 'paketobuildpacks/run@sha256:956341a43edf4029b29114502f7deb185527aed63e2a5ab94d86cd33abb34e56'
 > Pulling buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9:latest' ..................................................
 > Pulled buildpack image 'gcr.io/paketo-buildpacks/eclipse-openj9@sha256:9ec7ecf806716d1416b6f9030dea08a33cf1ed2d35dff5c2c4cab0d7f46cd4d1'
 > Executing lifecycle version v0.11.3
 > Using build cache volume 'pack-cache-4a387a19819.build'

 > Running creator
    [creator]     ===> DETECTING
    [creator]     6 of 19 buildpacks participating
    [creator]     paketo-buildpacks/eclipse-openj9    8.0.0
    [creator]     paketo-buildpacks/ca-certificates   2.2.0
    [creator]     paketo-buildpacks/bellsoft-liberica 8.0.0
    [creator]     paketo-buildpacks/executable-jar    5.0.0
    [creator]     paketo-buildpacks/dist-zip          4.0.0
    [creator]     paketo-buildpacks/spring-boot       4.3.0
    [creator]     ===> ANALYZING
    [creator]     Previous image with name "docker.io/library/build-demo:0.0.1-SNAPSHOT" not found
    [creator]     ===> RESTORING
    [creator]     ===> BUILDING
    [creator]
    [creator]     Paketo Eclipse OpenJ9 Buildpack 8.0.0
    [creator]       https://github.com/paketo-buildpacks/eclipse-openj9
    [creator]       Build Configuration:
    [creator]         $BP_JVM_VERSION              8.*             the Java version
    [creator]       Launch Configuration:
    [creator]         $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    [creator]         $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    [creator]         $JAVA_TOOL_OPTIONS                           the JVM launch flags
    [creator]       OpenJ9 JRE 8.0.292: Contributing to layer
    [creator]         Downloading from https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u292-b10_openj9-0.26.0/OpenJDK8U-jre_x64_linux_openj9_8u292b10_openj9-0.26.0.tar.gz
    [creator]         Verifying checksum
    [creator]         Expanding to /layers/paketo-buildpacks_eclipse-openj9/jre
    [creator]         Adding 129 container CA certificates to JVM truststore
    [creator]         Writing env.launch/BPI_APPLICATION_PATH.default
    [creator]         Writing env.launch/BPI_JVM_CACERTS.default
    [creator]         Writing env.launch/BPI_JVM_CLASS_COUNT.default
    [creator]         Writing env.launch/BPI_JVM_EXT_DIR.default
    [creator]         Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
    [creator]         Writing env.launch/JAVA_HOME.default
    [creator]         Writing env.launch/MALLOC_ARENA_MAX.default
...
    [creator]     Saving docker.io/library/build-demo:0.0.1-SNAPSHOT...
    [creator]     *** Images (c14b0f2252cd):
    [creator]           docker.io/library/build-demo:0.0.1-SNAPSHOT

Successfully built image 'docker.io/library/build-demo:0.0.1-SNAPSHOT'

I've seen the buildpack.toml: permission denied error before, but it's always been caused by real permissions problems on the file in the buildpack image and that doesn't seem to be the case here.

Can you share a minimal build.gradle file that reproduces the problem for you as well as the OS you are running Gradle on (Windows, MacOS, Linux?) and the Docker daemon you are using (Docker Desktop or something else)?

[bala-striva]

Sure, with a sample generated project with reactive web dependency I can reproduce the issue on my machine.

build.gradle

plugins {
	id 'org.springframework.boot' version '2.5.0'
	id 'io.spring.dependency-management' version '1.0.11.RELEASE'
	id 'java'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '11'

repositories {
	mavenCentral()
}

dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-webflux'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'io.projectreactor:reactor-test'
}

test {
	useJUnitPlatform()
}

bootBuildImage {
	buildpacks = [
	        'gcr.io/paketo-buildpacks/eclipse-openj9:latest',
		'paketo-buildpacks/java'
	]
}

OS:

Arch Linux 5.12.6-arch1-1 #1 SMP PREEMPT Sun, 23 May 2021 00:45:50 +0000 x86_64 GNU/Linux

Docker from arch linux package repository (https://archlinux.org/packages/community/x86_64/docker/):

Client:
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c28948e
 Built:             Mon Apr 12 14:10:41 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.3
  Git commit:       8728dd246c
  Built:            Mon Apr 12 14:10:25 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.2
  GitCommit:        36cc874494a56a253cd181a1a685b44b58a2e34a.m
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[bala-striva]

Using pack it seems to work on my setup.
I don't know if this is equivalent with running it from gradle.

at 09:10:40 ❯ pack build demo --path . --buildpack gcr.io/paketo-buildpacks/eclipse-openj9 --buildpack paketo-buildpacks/java --builder paketobuildpacks/builder:base
base: Pulling from paketobuildpacks/builder
Digest: sha256:148282212248cd49988c0cdddff19ee7091f02b03b2b4bce52e5d68640e7b0c5
Status: Image is up to date for paketobuildpacks/builder:base
base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:956341a43edf4029b29114502f7deb185527aed63e2a5ab94d86cd33abb34e56
Status: Image is up to date for paketobuildpacks/run:base-cnb
latest: Pulling from paketo-buildpacks/eclipse-openj9
Digest: sha256:9ec7ecf806716d1416b6f9030dea08a33cf1ed2d35dff5c2c4cab0d7f46cd4d1
Status: Image is up to date for gcr.io/paketo-buildpacks/eclipse-openj9:latest
===> DETECTING
8 of 19 buildpacks participating
paketo-buildpacks/eclipse-openj9    8.0.0
paketo-buildpacks/ca-certificates   2.2.0
paketo-buildpacks/bellsoft-liberica 8.0.0
paketo-buildpacks/gradle            5.2.0
paketo-buildpacks/executable-jar    5.0.0
paketo-buildpacks/apache-tomcat     5.4.0
paketo-buildpacks/dist-zip          4.0.0
paketo-buildpacks/spring-boot       4.3.0
===> ANALYZING
Previous image with name "demo" not found
===> RESTORING
===> BUILDING

Paketo Eclipse OpenJ9 Buildpack 8.0.0
  https://github.com/paketo-buildpacks/eclipse-openj9
  Build Configuration:
    $BP_JVM_VERSION              11              the Java version
  Launch Configuration:
    $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                           the JVM launch flags
  OpenJ9 JDK 11.0.11: Contributing to layer
    Downloading from https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.11%2B9_openj9-0.26.0/OpenJDK11U-jdk_x64_linux_openj9_11.0.11_9_openj9-0.26.0.tar.gz
    Verifying checksum
    Expanding to /layers/paketo-buildpacks_eclipse-openj9/jdk
    Adding 129 container CA certificates to JVM truststore
    Writing env.build/JAVA_HOME.override
    Writing env.build/JDK_HOME.override
  OpenJ9 JRE 11.0.11: Contributing to layer
    Downloading from https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.11%2B9_openj9-0.26.0/OpenJDK11U-jre_x64_linux_openj9_11.0.11_9_openj9-0.26.0.tar.gz
    Verifying checksum
    Expanding to /layers/paketo-buildpacks_eclipse-openj9/jre
    Adding 129 container CA certificates to JVM truststore
    Writing env.launch/BPI_APPLICATION_PATH.default
    Writing env.launch/BPI_JVM_CACERTS.default
    Writing env.launch/BPI_JVM_CLASS_COUNT.default
    Writing env.launch/BPI_JVM_SECURITY_PROVIDERS.default
    Writing env.launch/JAVA_HOME.default
    Writing env.launch/MALLOC_ARENA_MAX.default
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/active-processor-count
    Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/java-opts
    Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/link-local-dns
    Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/memory-calculator
    Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/openssl-certificate-loader
    Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/security-providers-configurer
    Creating /layers/paketo-buildpacks_eclipse-openj9/helper/exec.d/security-providers-classpath-9
  JVMKill Agent 1.16.0: Contributing to layer
    Downloading from https://github.com/cloudfoundry/jvmkill/releases/download/v1.16.0.RELEASE/jvmkill-1.16.0-RELEASE.so
    Verifying checksum
    Copying to /layers/paketo-buildpacks_eclipse-openj9/jvmkill
    Writing env.launch/JAVA_TOOL_OPTIONS.append
    Writing env.launch/JAVA_TOOL_OPTIONS.delim
  Java Security Properties: Contributing to layer
    Writing env.launch/JAVA_SECURITY_PROPERTIES.default
    Writing env.launch/JAVA_TOOL_OPTIONS.append
    Writing env.launch/JAVA_TOOL_OPTIONS.delim

Paketo CA Certificates Buildpack 2.2.0
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper

Paketo BellSoft Liberica Buildpack 8.0.0
  https://github.com/paketo-buildpacks/bellsoft-liberica
  Build Configuration:
    $BP_JVM_VERSION              11              the Java version
  Launch Configuration:
    $BPL_JVM_HEAD_ROOM           0               the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes  the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250             the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                           the JVM launch flags

Paketo Gradle Buildpack 5.2.0
  https://github.com/paketo-buildpacks/gradle
  Build Configuration:
    $BP_GRADLE_BUILD_ARGUMENTS  --no-daemon assemble  the arguments to pass to Gradle
    $BP_GRADLE_BUILT_ARTIFACT   build/libs/*.[jw]ar   the built application artifact explicitly.  Supersedes $BP_GRADLE_BUILT_MODULE
    $BP_GRADLE_BUILT_MODULE                           the module to find application artifact in
    Creating cache directory /home/cnb/.gradle
  Compiled Application: Contributing to layer
    Executing gradlew --no-daemon assemble
Downloading https://services.gradle.org/distributions/gradle-7.0.2-bin.zip
..........10%...........20%...........30%..........40%...........50%...........60%..........70%...........80%...........90%..........100%

Welcome to Gradle 7.0.2!

Here are the highlights of this release:
 - File system watching enabled by default
 - Support for running with and building Java 16 projects
 - Native support for Apple Silicon processors
 - Dependency catalog feature preview

For more details see https://docs.gradle.org/7.0.2/release-notes.html

To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.0.2/userguide/gradle_daemon.html#sec:disabling_the_daemon.
Daemon will be stopped at the end of the build 
> Task :compileJava
> Task :processResources
> Task :classes
> Task :bootJarMainClassName
> Task :bootJar
> Task :jar
> Task :assemble

BUILD SUCCESSFUL in 1m 7s
5 actionable tasks: 5 executed
  Removing source code

Paketo Executable JAR Buildpack 5.0.0
  https://github.com/paketo-buildpacks/executable-jar
  Class Path: Contributing to layer
    Writing env/CLASSPATH.delim
    Writing env/CLASSPATH.prepend
  Process types:
    executable-jar: java org.springframework.boot.loader.JarLauncher (direct)
    task:           java org.springframework.boot.loader.JarLauncher (direct)
    web:            java org.springframework.boot.loader.JarLauncher (direct)

Paketo Spring Boot Buildpack 4.3.0
  https://github.com/paketo-buildpacks/spring-boot
  Creating slices from layers index
    dependencies
    spring-boot-loader
    snapshot-dependencies
    application
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
  Spring Cloud Bindings 1.7.1: Contributing to layer
    Downloading from https://repo.spring.io/release/org/springframework/cloud/spring-cloud-bindings/1.7.1/spring-cloud-bindings-1.7.1.jar
    Verifying checksum
    Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
  Web Application Type: Contributing to layer
    Reactive web application detected
    Writing env.launch/BPL_JVM_THREAD_COUNT.default
  4 application slices
  Image labels:
    org.springframework.boot.version
===> EXPORTING
Adding layer 'paketo-buildpacks/eclipse-openj9:helper'
Adding layer 'paketo-buildpacks/eclipse-openj9:java-security-properties'
Adding layer 'paketo-buildpacks/eclipse-openj9:jre'
Adding layer 'paketo-buildpacks/eclipse-openj9:jvmkill'
Adding layer 'paketo-buildpacks/ca-certificates:helper'
Adding layer 'paketo-buildpacks/executable-jar:classpath'
Adding layer 'paketo-buildpacks/spring-boot:helper'
Adding layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
Adding layer 'paketo-buildpacks/spring-boot:web-application-type'
Adding 5/5 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving demo...
*** Images (f6158dd851da):
      demo
Adding cache layer 'paketo-buildpacks/eclipse-openj9:jdk'
Adding cache layer 'paketo-buildpacks/gradle:application'
Adding cache layer 'paketo-buildpacks/gradle:cache'
Successfully built image demo

[dmikusa]

@bala-striva Your pack command is slightly different. It's doing a source build. When you run with Gradle, it is using what was built locally. Can you try running your pack command again and setting -p to point to the JAR/WAR that you built?

Also, add -v to the pack command for verbose output & -e BP_DEBUG=true as well. The latter will turn up the logging output from the buildpacks. It'll be a lot of output but if you can capture it when the problem occurs it could provide some clues. Thanks

[bala-striva]

@dmikusa-pivotal

Running pack build demo --path ./build/libs/demo-0.0.1-SNAPSHOT.jar --buildpack gcr.io/paketo-buildpacks/eclipse-openj9 --buildpack paketo-buildpacks/java --builder paketobuildpacks/builder:base
is also working and produces the docker image.

Adding environment to build.gradle

bootBuildImage {
	buildpacks = [
	        'gcr.io/paketo-buildpacks/eclipse-openj9:latest',
		'paketo-buildpacks/java'
	]

	environment = ['BP_DEBUG': 'true']
}

did not produce any new log lines compared to whats added to the issue

[scottfrederick]

@bala-striva Thank you for the additional information and testing, it's very helpful. The only thing that stands out to me so far is the OS you are using. We test extensively on MacOS and Ubuntu, it's possible there is something slightly different about Arch Linux that is causing your problem. I'll try to get something set up to test that combination of OS, Java, and Gradle myself.

[scottfrederick]

I can reproduce this problem by testing in an Arch Linux container. I'll investigate further to find the root cause of why this works differently on Arch Linux than it does on other Linux distros.

[scottfrederick]

I've confirmed that this is a bug in the Spring Boot plugins. The same failure will happen with directory and tar-gzipped buildpacks. Fixes are in progress.

[scottfrederick]

@bala-striva The fix for this issue should be in a Spring Boot 2.5.1-SNAPSHOT build soon. I've tested and verified the fix on Arch Linux in a container environment, but if you get a chance to test it on your machine before the 2.5.1 release that would be very helpful.

[bala-striva]

@scottfrederick with the snapshot build i was able to build the docker image with custom buildpacks.

Thanks for the detailed feedbacks!