Encoding issue when upgrading to Quarkus 3 using Dockerfile.native-micro

[gian1200]

Describe the bug

When upgrading to Quarkus 3.7.3 from Quarkus 2 (2.16.12.Final), logs started having problems printing some chars (áéíóú). This happened when running in a Kubernetes server.

Dockerfile.native-micro doesn't seem to be defaulting to UTF-8. Is this expected?

Output on remote running pod (Dockerfile.native-micro):

bash-4.4$ locale charmap
ANSI_X3.4-1968

bash-4.4$ locale -a
C
C.utf8
POSIX

Expected behavior

At least chars with accent (áéíóú) should correctly appear in logs

Actual behavior

Chars with accent (áéíóú) appear as "?" in the log

How to Reproduce?

Image used to build: ghcr.io/graalvm/graalvm-ce:ol8-java17-22

Steps to build:

1. gu install native-image
2. ./mvnw -B package -Dnative -Dmaven.test.skip

Output of uname -a or ver

No response

Output of java -version

17.0.8+7-jvmci-22.3-b22

Mandrel or GraalVM version (if different from Java)

GraalVM 22.3.3 Java 17 CE

Quarkus version or git rev

3.7.3

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.6

Additional information

Same image/steps to build worked fine with Quarkus 2

Maybe some related issues:
#11222
quarkusio/quarkus-images#86

Fragment build log:

...
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building native image source jar: /workspace/71bbac2e-06fd-4bda-8b47-4bfc7bd331be/target/<edited-app-name>-1.0.0-SNAPSHOT-native-image-source-jar/<edited-app-name>-1.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from /workspace/71bbac2e-06fd-4bda-8b47-4bfc7bd331be/target/<edited-app-name>-1.0.0-SNAPSHOT-native-image-source-jar/<edited-app-name>-1.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 22.3.3 JDK 17.0.8+7-jvmci-22.3-b22
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /opt/graalvm-ce-java17-22.3.3/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dlogging.initial-configurator.min-level=500 -J-Dio.quarkus.caffeine.graalvm.recordStats=true -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.hibernate.orm.runtime.graal.DisableLoggingFeature,org.hibernate.graalvm.internal.GraalVMStaticFeature,io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature,io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature,io.quarkus.jdbc.postgresql.runtime.graal.SQLXMLFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+CollectImageBuildStatistics -H:ImageBuildStatisticsFile=<edited-app-name>-1.0.0-SNAPSHOT-runner-timing-stats.json -H:BuildOutputJSONFile=<edited-app-name>-1.0.0-SNAPSHOT-runner-build-output-stats.json -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+ReportExceptionStackTraces -H:-AddAllCharsets --enable-url-protocols=http,https -H:NativeLinkerOption=-no-pie --enable-monitoring=heapdump -H:-UseServiceLoaderFeature -H:+StackTrace -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json <edited-app-name>-1.0.0-SNAPSHOT-runner -jar <edited-app-name>-1.0.0-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating '<edited-app-name>-1.0.0-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/7] Initializing...                                                                                   (25.4s @ 0.52GB)
 Version info: 'GraalVM 22.3.3 Java 17 CE'
 Java version info: '17.0.8+7-jvmci-22.3-b22'
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC
 8 user-specific feature(s)
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.caffeine.runtime.graal.CacheConstructorsFeature
 - io.quarkus.hibernate.orm.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - io.quarkus.jdbc.postgresql.runtime.graal.SQLXMLFeature
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
 - org.hibernate.graalvm.internal.GraalVMStaticFeature: Hibernate ORM's static reflection registrations for GraalVM
17:53:31,534 WARN  [org.hib.orm.deprecation] HHH90000025: PostgreSQLDialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)
[2/7] Performing analysis...  [*************]                                                          (187.0s @ 2.14GB)
  22,782 (91.35%) of 24,940 classes reachable
  32,503 (63.48%) of 51,204 fields reachable
 111,958 (54.04%) of 207,177 methods reachable
     919 classes,   442 fields, and 5,487 methods registered for reflection
      63 classes,    68 fields, and    55 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/7] Building universe...                                                                              (21.6s @ 2.85GB)
[4/7] Parsing methods...      [****]                                                                    (15.1s @ 2.52GB)
[5/7] Inlining methods...     [***]                                                                      (8.3s @ 2.33GB)
[6/7] Compiling methods...    [**********]                                                             (108.1s @ 2.85GB)
[7/7] Creating image...                                                                                  (9.0s @ 2.95GB)
  40.95MB (48.96%) for code area:    74,027 compilation units
  42.37MB (50.66%) for image heap:  501,746 objects and 24 resources
 325.91KB ( 0.38%) for other data
  83.63MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 packages in code area:                               Top 10 object types in image heap:
   1.64MB sun.security.ssl                                     9.25MB byte[] for code metadata
   1.16MB java.util                                            5.71MB java.lang.Class
 840.74KB io.quarkus.runtime.generated                         4.81MB byte[] for general heap data
 764.25KB java.lang.invoke                                     4.59MB java.lang.String
 717.68KB com.sun.crypto.provider                              4.39MB byte[] for java.lang.String
 533.40KB jdk.proxy4                                           1.91MB com.oracle.svm.core.hub.DynamicHubCompanion
 514.90KB io.netty.buffer                                      1.31MB byte[] for reflection metadata
 508.97KB java.lang                                          923.16KB java.lang.String[]
 491.57KB java.util.concurrent                               794.10KB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
 488.04KB org.postgresql.jdbc                                786.80KB java.util.HashMap$Node
  32.84MB for 931 more packages                                7.83MB for 5037 more object types
------------------------------------------------------------------------------------------------------------------------
                       39.7s (10.3% of total time) in 81 GCs | Peak RSS: 5.10GB | CPU load: 1.81
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /workspace/71bbac2e-06fd-4bda-8b47-4bfc7bd331be/target/<edited-app-name>-1.0.0-SNAPSHOT-native-image-source-jar/<edited-app-name>-1.0.0-SNAPSHOT-runner (executable)
 /workspace/71bbac2e-06fd-4bda-8b47-4bfc7bd331be/target/<edited-app-name>-1.0.0-SNAPSHOT-native-image-source-jar/<edited-app-name>-1.0.0-SNAPSHOT-runner-build-output-stats.json (json)
 /workspace/71bbac2e-06fd-4bda-8b47-4bfc7bd331be/target/<edited-app-name>-1.0.0-SNAPSHOT-native-image-source-jar/<edited-app-name>-1.0.0-SNAPSHOT-runner-timing-stats.json (raw)
 /workspace/71bbac2e-06fd-4bda-8b47-4bfc7bd331be/target/<edited-app-name>-1.0.0-SNAPSHOT-native-image-source-jar/<edited-app-name>-1.0.0-SNAPSHOT-runner.build_artifacts.txt (txt)
========================================================================================================================
Finished generating '<edited-app-name>-1.0.0-SNAPSHOT-runner' in 6m 25s.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] objcopy --strip-debug <edited-app-name>-1.0.0-SNAPSHOT-runner
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 394360ms
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:57 min
...

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @geoand (kubernetes), @iocanel (kubernetes), @zakkak (mandrel,native-image)

[mschorsch]

You can try to set the file encoding in native explicitly via quarkus.native.file-encoding.

https://quarkus.io/guides/building-native-image#quarkus-native-pkg-native-config_quarkus-native-file-encoding

Defines the file encoding as in -Dfile.encoding=…​. Native image runtime uses the host’s (i.e. build time) value of file.encoding system property. We intentionally default this to UTF-8 to avoid platform specific defaults to be picked up which can then result in inconsistent behavior in the generated native executable.

[mschorsch]

When building the native image, you can include the following properties in your application.properties file:

quarkus.native.file-encoding=UTF-8
quarkus.native.add-all-charsets=true

[gian1200]

I'm going to test your suggestion. However, I'm not sure if it will work.

In the logs, you can see that -J-Dfile.encoding=UTF-8 and -H:-AddAllCharsets are already set.

I thought that the fix was going to be in the lines of making Dockerfile.native-micro UTF-8 by default.

Will update the issue shortly with the results.

[gian1200]

Update: It didn't work

[gian1200]

For the sake of testing, also I tried setting the language ENVs to UTF-8 in the Dockerfile, without success

• Dockerfile.native-micro:

FROM quay.io/quarkus/quarkus-micro-image:2.0

ENV LANGUAGE='en_US:en'
ENV LANG='C.UTF-8'
ENV LC_ALL='C.UTF-8'

WORKDIR /work/
RUN chown 1001 /work \
    && chmod "g+rwX" /work \
    && chown 1001:root /work
COPY --chown=1001:root target/*-runner /work/application

EXPOSE 8080
USER 1001

ENTRYPOINT ["./application", "-Dquarkus.http.host=0.0.0.0"]

• Terminal:

bash-4.4$ locale -a
C
C.utf8
POSIX

bash-4.4$ locale
LANG=C.UTF-8
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=C.UTF-8

bash-4.4$ locale charmap
UTF-8

[galderz]

@gian1200 Have you tried the latest Quarkus version? Today that is 3.9.3. If the issue still persists, can you be more specific on the reproducer?

I did the following and it all looks fine with the Quarkus getting-started-reactive:

1. Added import io.quarkus.logging.Log; to ReactiveGreetingResource and then added Log.info("áéíóú"); before returning hello in hello() method.

2. On a mac, so did an in-container build to get a Linux binary:

$ ./mvnw package -Dnative -Dquarkus.native.container-build

3. Built a container with:

$ podman build -f src/main/docker/Dockerfile.native-micro -t quarkus/getting-started-reactive .

4. Run the container with:

$ podman run -i --rm -p 8080:8080 quarkus/getting-started-reactive

5. Did a curl on /hello and the logs print:

2024-04-17 11:26:04,510 INFO  [org.acm.get.sta.ReactiveGreetingResource] (vert.x-eventloop-thread-0) áéíóú

[gian1200]

Thanks for answering, @galderz. I'll try to update it and retry.

The only difference I see from your reproducer is that we use ghcr.io/graalvm/graalvm-ce:ol8-java17-22 as base image for building (we are unable to run ./mvnw package -Dnative -Dquarkus.native.container-build on CI env). If I'm not wrong, that command runs Mandrel, correct?

[zakkak]

A couple of hours ago oracle/graal#8792 was reported which seems/could be related.

[galderz]

we are unable to run ./mvnw package -Dnative -Dquarkus.native.container-build on CI env

What issues are you having in your CI env?

Underneath -Dquarkus.native.container-build fires of an in-container build using the default Mandrel container build image which is quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21. We also have ubi-quarkus-graalvmce-builder-image images here. I guess you could try with one of our Mandrel or GraalVM CE images?

[gian1200]

What issues are you having in your CI env?

No need to worry about it. It's an issue specific to our CI env. We don't have docker or podman available inside the containers where our pipelines are running.

I guess you could try with one of our Mandrel or GraalVM CE images?

Yeah, I was actually doing that (still WIP). I will also try with ol7 to confirm if I am affected by oracle/graal#8792 or not (oracle/graal#8792 (comment)).

I will post an update in a few days.

[gian1200]

Quick update: confirmed. It worked with ol7. This means this is affected by oracle/graal#8792

[gian1200]

Since the issue is in Graal and not Quarkus, I'm proceeding to close the issue.

PS: I didn't test it with ubi-quarkus-native-image:22.3-java17, so I'm not sure if there is also an issue there.