bug: S3 integration returns nonconformant ListVersionsResult object as per provided AmazonS3.xsd

[Dragas]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Returned ListVersionsResult element for bucket versions request does not conform to the expected structure provided by amazon s3.

Expected Behavior

ListVersionsResult element matches that provided by actual s3.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

Following is fragment from docker-compose

      localstack:
         image: localstack/localstack:3.4
         ports:
          - "4566:4566"
          - "4510-4559:4510-4559"
         environment:
          - SERVICES=s3
          - DOCKER_HOST=unix:///var/run/docker.sock
          - AWS_DEFAULT_REGION=us-east-1
         volumes:
          - /var/run/docker.sock:/var/run/docker.sock

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Setup the created bucket somewhat close to the awsdocs bucket provided by amazon.

        touch arbitraryfile.txt
        awslocal s3 mb s3://awsdocs
        awslocal s3api put-object --bucket awsdocs --key "AMP/2.0/configuring-connector-2-0.pdf" --body arbitraryfile.txt
        awslocal s3api put-object --bucket awsdocs --key "AMP/2.0/deploying-connector-2-0.pdf" --body arbitraryfile.txt
        awslocal s3api put-object --bucket awsdocs --key "AMP/2.0/managing-connector-2-0.pdf" --body arbitraryfile.txt
        awslocal s3api put-object --bucket awsdocs --key "AWSCloudFormation/2010-05-15/cfn-api-2010-05-15.pdf" --body arbitraryfile.txt
        awslocal s3api put-object --bucket awsdocs --key "AWSCloudFormation/2010-05-15/cfn-apiref-2010-05-15.pdf" --body arbitraryfile.txt
        awslocal s3api put-object --bucket awsdocs --key "AWSCloudFormation/2010-05-15/cfn-gsg-2010-05-15.pdf" --body arbitraryfile.txt\
        awslocal s3api put-object --bucket awsdocs --key "AWSCloudFormation/2010-05-15/cfn-ug-2010-05-15.pdf" --body arbitraryfile.txt
        awslocal s3api put-object --bucket awsdocs --key "AWSCloudFormation/2010-05-15/cfn-ug-cli-ref-09172013.pdf" --body arbitraryfile.txt

Reproduction test (in java)

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.example</groupId>
    <artifactId>reproduction</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.12.716</version>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

S3IntegrationTest.java

package com.ex;

import java.util.Collection;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.ListVersionsRequest;
import com.amazonaws.services.s3.model.S3VersionSummary;
import com.amazonaws.services.s3.model.VersionListing;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.AnonymousAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.BucketVersioningConfiguration;
import com.amazonaws.services.s3.model.SetBucketVersioningConfigurationRequest;

class S3IntegrationTest {
    private void performTest(String endpointUri) {
        AwsClientBuilder.EndpointConfiguration endpoint = new AwsClientBuilder.EndpointConfiguration(endpointUri,
                "us-east-1");
        AmazonS3 client = AmazonS3ClientBuilder
                .standard()
                .withPathStyleAccessEnabled(true)
                .withEndpointConfiguration(endpoint)
                .withCredentials(new AWSStaticCredentialsProvider(new AnonymousAWSCredentials()))
                .build();
        ListVersionsRequest request = new ListVersionsRequest()
                .withBucketName("awsdocs")
                .withMaxResults(10);
        VersionListing response = client.listVersions(request);
        Collection<S3VersionSummary> summaries = response.getVersionSummaries();
        for (S3VersionSummary it : summaries) {
            Assertions.assertNotNull(it.getBucketName());
        }
    }
    @Test
    public void referenceIntegrationTest() {
        performTest("https://s3.amazonaws.com/");
    }
    @Test
    public void localstackIntegrationTest() {
        performTest("http://localhost:4566/"); // fails on 3.4, but passes on 0.12.16
    }
}

Environment

- OS: Mac os sonoma 14.4.1 (localstack runs in docker)
- LocalStack: 3.4

Anything else?

Only reproducable with java aws s3 sdk v1 because in v2 the API is different and no longer provides the bucket name reference on the versions summary entity. Seems to have worked as intended in 0.12.16.

Reference XSD: https://doc.s3.amazonaws.com/2006-03-01/AmazonS3.xsd

Reference bucket that provides versions: https://s3.amazonaws.com/awsdocs?versions

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[bentsku]

Hello @Dragas and thanks for your report! I'll work on it shortly. This is an interesting bug, because it seems the AWS Java SDK v1 rely on the order of the XML fields to parse the response. We do indeed return the right XML fields, but apparently not in the right order for the custom parsing of the v1 SDK.

We've had the same issue for the Owner field in ListBuckets, you can see the fix here: #8329

I'll start working on a fix and update you here once it's available.

[Dragas]

Yeah, I think it would be best to match the structure thats provided by xsd, because it specifies sequences which expects explicit element order.

…

On Fri, May 10, 2024, 14:40 Ben Simon Hartung ***@***.***> wrote: Hello @Dragas <https://github.com/Dragas> and thanks for your report! I'll work on it shortly. This is an interesting bug, because it seems the AWS Java SDK v1 rely on the order of the XML fields to parse the response. We've had the same issue for the Owner field in ListBuckets, you can see the fix here: #8329 <#8329> I'll start working on a fix and update you here once it's available. — Reply to this email directly, view it on GitHub <#10801 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOMALBC5UWQB4Y5HI72FCDZBSW33AVCNFSM6AAAAABHPC3ET6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBUGQ3DAMJXGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>