AWSTraceHeader - getting error as Duplicated keys are provided for SQS

[ssutar-apptio]

Hi there,

Getting following error in 1.12.261 onwards when there is implementation for "Attribute" addition is done for SQS.

The following error occurs when "AWSTraceHeader" is not enabled in AWS, but we use Datadog and it automatically tries to add "AWSTraceHeader".

java.lang.IllegalArgumentException: Duplicated keys (AWSTraceHeader) are provided. at com.amazonaws.services.sqs.model.Message.addAttributesEntry(Message.java:599) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master] at com.amazonaws.services.sqs.model.transform.MessageStaxUnmarshaller.unmarshall(MessageStaxUnmarshaller.java:152) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master] at com.amazonaws.services.sqs.model.transform.ReceiveMessageResultStaxUnmarshaller.unmarshall(ReceiveMessageResultStaxUnmarshaller.java:47) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master] at com.amazonaws.services.sqs.model.transform.ReceiveMessageResultStaxUnmarshaller.unmarshall(ReceiveMessageResultStaxUnmarshaller.java:28) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master] at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:117) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master] at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:42) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master] at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794) ~[app.jar:1.0.12-SNAPSHOT-444-branch-master]

Can you please have a look by integrating Datadog with the aws-sdk-java?

Thanks,
Sunil Sutar

[ssutar-apptio]

This is the full expanded stack trace from Datadog -

com.amazonaws.SdkClientException: Unable to unmarshall response (Duplicated keys (AWSTraceHeader) are provided.). Response Code: 200, Response Text: OK at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1818) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2329) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2296) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2285) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1715) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1683) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.cloudability.commons.aws.messaging.SQSFairConsumerService.waitForMessage(SQSFairConsumerService.java:233) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.cloudability.commons.aws.messaging.SQSFairConsumerService.run(SQSFairConsumerService.java:196) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.cloudability.commons.aws.messaging.SQSFairConsumerService.lambda$startUp$0(SQSFairConsumerService.java:162) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.IllegalArgumentException: Duplicated keys (AWSTraceHeader) are provided. at com.amazonaws.services.sqs.model.Message.addAttributesEntry(Message.java:599) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.model.transform.MessageStaxUnmarshaller.unmarshall(MessageStaxUnmarshaller.java:152) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.model.transform.ReceiveMessageResultStaxUnmarshaller.unmarshall(ReceiveMessageResultStaxUnmarshaller.java:47) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.services.sqs.model.transform.ReceiveMessageResultStaxUnmarshaller.unmarshall(ReceiveMessageResultStaxUnmarshaller.java:28) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:117) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:42) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69) ~[app.jar:1.76-SNAPSHOT-355-branch-master] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794) ~[app.jar:1.76-SNAPSHOT-355-branch-master] ... 23 more

[debora-ito]

Can you please have a look by integrating Datadog with the aws-sdk-java?

@ssutar-apptio I'm sorry but issues with third-party integrations are out of scope of the Java SDK. We guarantee that the SDK will work when it communicates directly with aws endpoints, we don't guarantee when third-party solutions are involved. You should reach out to the Datadog support team.

[ssutar-apptio]

I understood and reached to Datadog support too.

They don't add the header by themselves, they just use it if it's there. And I received the below reply ->

`Hi Sunil,

Thanks for sharing all these details.

We believe that Java tracer doesn't directly add AWSTraceHeader to messages. It only adds the current trace id to AWS requests in the form of a X-Amzn-Trace-Id HTTP header (we use this approach because most AWS Java requests are signed, so we can't add arbitrary headers). This is then converted to AWSTraceHeader by AWS itself when it processes the client request, which the tracer will read on the other side of the queue when consuming messages - but we don't add AWSTraceHeader directly, we only read it if it's there.

Could you also try using DD_AWS_SDK_PROPAGATION_ENABLED=false to turn off the addition of the X-Amzn-Trace-Id HTTP header - that would then help us at least rule out the AWSTraceHeader added by AWS, because we'd no longer be adding X-Amzn-Trace-Id HTTP headers to Java client requests But you would then be responsible in propagating the trace ids. Would this be an option that you can possibly try?

For context, this is because the use of X-Amzn-Trace-Id HTTP headers and extraction of AWSTraceHeader attributes has been in the Java tracer for a while now. The only recent change was related to when we create spans, not how propagation occurs.

Thank you and looking forward to hear back from you.
Regards,
Jimmy Prasetyo | Technical Support Engineer | Datadog
`

Could you please check if it's the issue on aws-sdk side? We are not getting this error if I disable trace from DD side. Today I am going to try this solution of DD_AWS_SDK_PROPAGATION_ENABLED=false to see if it helps to identify if the issue is on aws-sdk side.

Thank you,
Sunil Sutar