Direct memory leak when getting object metadata inside parallelStream with Java 17

[winzsanchez]

Describe the bug

When calling client.getObjectMetadata(bucket, key) inside a parallelStream(), we notice the direct memory usage going up when using Java 17 (and there are multiple objects in the bucket).

This does not happen with Java 11 or when using stream().

Expected Behavior

Direct memory shouldn't be going up.

Current Behavior

When using Java 17 we see the following:

[MEM] Initial memory used: 16384
[MEM] Memory used after serial stream: 16384
[MEM] Memory used after parallel stream: 57344
[MEM] Memory used after serial stream: 57344

With Java 11:

[MEM] Initial memory used: 8192
[MEM] Memory used after serial stream: 8192
[MEM] Memory used after parallel stream: 8192
[MEM] Memory used after serial stream: 8192

Reproduction Steps

    protected static BufferPoolMXBean directMemoryMXBean;

    static {
        for (final BufferPoolMXBean pool : ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class)) {
            if (pool.getName().equals("direct")) {
                directMemoryMXBean = pool;
            }
        }
    }

    public static void listObjects(AmazonS3 client, String bucketName, String prefix) throws InterruptedException {
        ObjectListing listing = client.listObjects( bucketName, prefix );
        var objectSummaries = listing.getObjectSummaries();

        while (listing.isTruncated()) {
            listing = client.listNextBatchOfObjects (listing);
            objectSummaries.addAll (listing.getObjectSummaries());
        }
        System.out.println("objectSummaries: " + objectSummaries);

        System.out.println("[MEM] Initial memory used: " + directMemoryMXBean.getMemoryUsed());
        objectSummaries.forEach(printMetadata(client, bucketName));
        System.out.println("[MEM] Memory used after serial stream: " + directMemoryMXBean.getMemoryUsed());

        objectSummaries.parallelStream().forEach(printMetadata(client, bucketName));
        System.out.println("[MEM] Memory used after parallel stream: " + directMemoryMXBean.getMemoryUsed());

        objectSummaries.forEach(printMetadata(client, bucketName));
        System.out.println("[MEM] Memory used after serial stream: " + directMemoryMXBean.getMemoryUsed());
    }

    private static Consumer<S3ObjectSummary> printMetadata(AmazonS3 client, String bucketName) {
        return s -> System.out.println(s + " metadata: " + getMetadata(client, bucketName, s.getKey()));
    }

    public static ObjectMetadata getMetadata(AmazonS3 client, String bucketName, String name) {
        return client.getObjectMetadata(bucketName, name);
    }

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

1.12.606

JDK version used

java version "17.0.9" 2023-10-17 LTS Java(TM) SE Runtime Environment (build 17.0.9+11-LTS-201) Java HotSpot(TM) 64-Bit Server VM (build 17.0.9+11-LTS-201, mixed mode, sharing)

Operating System and version

Windows 10

[debora-ito]

@winzsanchez can you generate the SDK client-side metrics? We do generate metrics about used memory, it would be interesting to see a comparison of the cases side by side.

For instructions on how to generate the client-side metrics please check our Developer Guide.
Also check our blog post that shows how to interpret the metrics: Tuning the AWS SDK for Java to Improve Resiliency
- this is mostly about timeouts and retries, not so much on memory usage, but it's informative.

[winzsanchez]

JvmMetric-2023_12_18_10_45_00-2023_12_18_11_20_00-UTC-5.csv

Hi @debora-ito, attached are the memory metrics.
The ones from 10:45 to 10:55 are with Java 11.
11:05 to 11:20 are with Java 17