S3EventNotification can't be deserialized

[jeusdi]

Type: Bug

Component:
"S3", "SQS"

Describe the bug
I'm really facing up with S3 event lambda function deserialization.

I'm getting:

ClassCastException: class org.springframework.util.LinkedMultiValueMap cannot be cast to class com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification

Detailed trace:

ClassCastException: class org.springframework.util.LinkedMultiValueMap cannot be cast to class com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification (org.springframework.util.LinkedMultiValueMap and com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification are in unnamed module of loader 'app')] with root cause

java.lang.ClassCastException: class org.springframework.util.LinkedMultiValueMap cannot be cast to class com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification (org.springframework.util.LinkedMultiValueMap and com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification are in unnamed module of loader 'app')
        at net.gencat.transversal.espaidoc.presentation.backoffice.apigateway.document.pushed.PushedDocumentLambdaConsumer.apply(PushedDocumentLambdaConsumer.java:17) ~[classes/:na]
        at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunctionAndEnrichResultIfNecessary(SimpleFunctionRegistry.java:958) ~[spring-cloud-function-context-4.1.0.jar:4.1.0]
        at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunction(SimpleFunctionRegistry.java:904) ~[spring-cloud-function-context-4.1.0.jar:4.1.0]
        at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:740) ~[spring-cloud-function-context-4.1.0.jar:4.1.0]
        at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:580) ~[spring-cloud-function-context-4.1.0.jar:4.1.0]
        at org.springframework.cloud.function.web.util.FunctionWebRequestProcessingHelper.processRequest(FunctionWebRequestProcessingHelper.java:132) ~[spring-cloud-function-web-4.1.0.jar:4.1.0]
        at org.springframework.cloud.function.web.mvc.FunctionController.form(FunctionController.java:98) ~[spring-cloud-function-web-4.1.0.jar:4.1.0]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:259) ~[spring-web-6.1.4.jar:6.1.4]

Sample
Here you can find code repo.

I'm able to reach my function since I've added spring-cloud-starter-function-web dependency.

Then, here my test s3 notification:

{
   "Records":[
      {
         "body":"{\"Records\": [{\"eventVersion\": \"2.1\", \"eventSource\": \"aws:s3\", \"awsRegion\": \"us-east-1\", \"eventTime\": \"2024-02-28T05:30:34.882Z\", \"eventName\": \"ObjectCreated:Put\", \"userIdentity\": {\"principalId\": \"AIDAJDPLRKLG7UEXAMPLE\"}, \"requestParameters\": {\"sourceIPAddress\": \"127.0.0.1\"}, \"responseElements\": {\"x-amz-request-id\": \"b38d4b33\", \"x-amz-id-2\": \"eftixk72aD6Ap51TnqcoF8eFidJG9Z/2\"}, \"s3\": {\"s3SchemaVersion\": \"1.0\", \"configurationId\": \"058322eb\", \"bucket\": {\"name\": \"espaidoc\", \"ownerIdentity\": {\"principalId\": \"A3NL1KOZZKExample\"}, \"arn\": \"arn:aws:s3:::espaidoc\"}, \"object\": {\"key\": \"references/reference1.readme\", \"sequencer\": \"0055AED6DCD90281E5\", \"size\": 1917, \"eTag\": \"e8b5cd0ccb83955551ec584503cc4bd4\"}}}]}",
         "receiptHandle":"ZWQ1YmRjYjctODU0OC00NzMyLWIzNWEtOWMyZjRjMjk4MzdjIGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6U3FzUXVldWVDcmVhdGUgMDdlYWZlMTQtOTRjMi00YjA5LTgwNjMtNTU2M2MyOTViMmFjIDE3MDkwOTgyMzUuODg5MDgwOA==",
         "md5OfBody":"dff5137ad0700f8f44763d5eed490032",
         "eventSourceARN":"arn:aws:sqs:us-east-1:000000000000:SqsQueueCreate",
         "eventSource":"aws:sqs",
         "awsRegion":"us-east-1",
         "messageId":"07eafe14-94c2-4b09-8063-5563c295b2ac",
         "attributes":{
            "SenderId":"000000000000",
            "SentTimestamp":"1709098234899",
            "ApproximateReceiveCount":"1",
            "ApproximateFirstReceiveTimestamp":"1709098235889"
         },
         "messageAttributes":{

         }
      }
   ]
}

In order to reach my function via http I'm using:

curl -i localhost:8080/pushedDocumentConsumer --data @s3-event.json

My Consumer is:

@Component
@RequiredArgsConstructor
@Slf4j
public class PushedDocumentLambdaConsumer implements Function<S3EventNotification, String> {

	@Override
	public String apply(S3EventNotification s3EventNotification) {
		log.info("************************************************* RECEIVED TOSTRING()");
		log.info(s3EventNotification.toString());
		log.info("*************************************************");

		log.info("processing event");
		if (s3EventNotification != null && s3EventNotification.getRecords() != null) {
			log.info("Number of records: {}", s3EventNotification.getRecords().size());
			for (S3EventNotificationRecord s3EventNotificationRecord : s3EventNotification.getRecords()) {
				log.info("event name: {}", s3EventNotificationRecord.getEventName());
				log.info("event source: {}", s3EventNotificationRecord.getEventSource());

				S3Entity s3Entity = s3EventNotificationRecord.getS3();
				if (null != s3Entity) {
					log.info("s3Entity bucket name: {}", s3Entity.getBucket().getName());
					log.info("s3Entity object name: {}", s3Entity.getObject().getKey());
				}
			}
		}

		log.info("======================");
		return "Ok";
	}

My pom.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>net.gencat.transversal.espaidoc</groupId>
	<artifactId>backoffice-apigateway</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>backoffice-apigateway</name>
	<description>espaidoc backoffice apigateway</description>

	<properties>
		<maven.compiler.target>17</maven.compiler.target>
    <maven.compiler.source>17</maven.compiler.source>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

		<!-- <spring-boot-dependencies.version>3.2.2</spring-boot-dependencies.version> -->
		<spring-cloud-dependencies.version>2023.0.0</spring-cloud-dependencies.version>
    <spring-cloud-aws.version>3.1.0</spring-cloud-aws.version>
		<spring-cloud-function.version>4.1.0</spring-cloud-function.version>
		<thin-wrapper-layout.version>1.0.31.RELEASE</thin-wrapper-layout.version>

		<aws-lambda-events.version>3.11.4</aws-lambda-events.version>
		<aws-lambda-java-core.version>1.2.3</aws-lambda-java-core.version>
		<aws-lambda-java-serialization.version>1.1.5</aws-lambda-java-serialization.version>

		<jackson.version>2.16.1</jackson.version>
	</properties>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>3.2.3</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-function-dependencies</artifactId>
				<version>${spring-cloud-function.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
			<dependency>
				<groupId>com.amazonaws</groupId>
				<artifactId>aws-java-sdk-bom</artifactId>
				<version>1.12.660</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<dependencies>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-function-adapter-aws</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-function-web</artifactId>
		</dependency>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-lambda-java-events</artifactId>
			<version>${aws-lambda-events.version}</version>
		</dependency>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-lambda-java-core</artifactId>
			<version>${aws-lambda-java-core.version}</version>
		</dependency>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-lambda-java-serialization</artifactId>
			<version>${aws-lambda-java-serialization.version}</version>
		</dependency>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-java-sdk-s3</artifactId>
		</dependency>
		<dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk-sqs</artifactId>
    </dependency>

		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<version>1.18.30</version>
			<scope>provided</scope>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-deploy-plugin</artifactId>
				<configuration>
					<skip>true</skip>
				</configuration>
			</plugin>
			<!-- <plugin> -->
			<!-- 	<groupId>org.springframework.boot</groupId> -->
			<!-- 	<artifactId>spring-boot-maven-plugin</artifactId> -->
			<!-- 	<dependencies> -->
			<!-- 		<dependency> -->
			<!-- 			<groupId>org.springframework.boot.experimental</groupId> -->
			<!-- 			<artifactId>spring-boot-thin-layout</artifactId> -->
			<!-- 			<version>${thin-wrapper-layout.version}</version> -->
			<!-- 		</dependency> -->
			<!-- 	</dependencies> -->
			<!-- </plugin> -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-shade-plugin</artifactId>
				<configuration>
					<createDependencyReducedPom>false</createDependencyReducedPom>
					<shadedArtifactAttached>true</shadedArtifactAttached>
					<shadedClassifierName>aws</shadedClassifierName>

				</configuration>
			</plugin>
		</plugins>
	</build>
</project>

[garytxo]

Hi Jordi, I am no expert but it looks to me that you are using io.awspring.cloud version 3.1.0 which by default uses on AWS JDK 2.0 as illustrated :

Then I see a reference to AWS JDK version 1.2.3 in the pom which could be problematic and I imagine this dependency was added to reference the S3EventNotification which is currently not available in AWS JDK 2.0.
There is a ticket open requesting to migrate this object to SDK 2.X but it has not be done yet however, some people have provided workarounds which may solve your problem.

[edelauna]

Looks like the issue has been closed, I think bumping the amazon dependencies should help and prevent having to include a separate version of awssdk: #1115