Downloading Directory with SSE KMS encryption from S3 bucket fails #2995

rickyeng127 · 2023-06-28T12:40:07Z

Describe the bug

When using the example code to download all objects in a bucket that is encrypted with a SSE KMS key, I am seeing the following exception:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: S3, Status Code: 403, Request ID: X7WTMWZZ7Y99AKZ7, Extended Request ID: 82DqRygu2rM4uHTHXI2Vs9ddWWVyq0kPHie53MQbuWYoyUElX/y5YNBO7C8GlIs2cq0L/gNITgo=)

The example code that I am using is:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_DownloadBucketToDirectory_section.html

Expected Behavior

I am expecting to see all all the objects in a bucket downloaded to the specified local directory.

Current Behavior

An error is returned with full stacktrace:

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send request

at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
at com.fanniemae.prism.util.aws.S3Util.downloadFilesToDirectory(S3Util.java:440)
at com.fanniemae.prism.test.S3.TestS3.testDownloadDirectory(TestS3.java:304)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send request
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at software.amazon.awssdk.transfer.s3.internal.DownloadDirectoryHelper.lambda$doDownloadDirectory$2(DownloadDirectoryHelper.java:121)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.transfer.s3.internal.AsyncBufferingSubscriber.onError(AsyncBufferingSubscriber.java:79)
at software.amazon.awssdk.utils.async.DelegatingSubscriber.onError(DelegatingSubscriber.java:40)
at software.amazon.awssdk.core.internal.pagination.async.ItemsSubscription.lambda$fetchNextPage$0(ItemsSubscription.java:89)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:54)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:67)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:170)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:105)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:238)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: S3, Status Code: 403, Request ID: X7WTMWZZ7Y99AKZ7, Extended Request ID: 82DqRygu2rM4uHTHXI2Vs9ddWWVyq0kPHie53MQbuWYoyUElX/y5YNBO7C8GlIs2cq0L/gNITgo=)
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
... 27 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: S3, Status Code: 403, Request ID: X7WTMWZZ7Y99AKZ7, Extended Request ID: 82DqRygu2rM4uHTHXI2Vs9ddWWVyq0kPHie53MQbuWYoyUElX/y5YNBO7C8GlIs2cq0L/gNITgo=)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:719)
at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:731)
at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2023)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:135)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:124)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)

Reproduction Steps

This is the code I am using to download the files in the bucket:

public void downloadFilesToDirectory(String destinationPath, String prefix) throws InterruptedException {

  S3AsyncClient s3AsyncClient =
    S3AsyncClient.crtBuilder()
      .targetThroughputInGbps(20.0)
      .minimumPartSizeInBytes(1000000L)
      .build();

S3TransferManager transferManager = S3TransferManager.builder()
    .s3Client(s3AsyncClient)
    .build();

DownloadDirectoryRequest downloadDirectoryRequest = DownloadDirectoryRequest.builder()
  .destination(Paths.get(destinationPath))
  .bucket(this.s3BucketName + "/" + prefix)
  .build();

DirectoryDownload directoryDownload = transferManager.downloadDirectory(downloadDirectoryRequest);

CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

completedDirectoryDownload.failedTransfers().forEach(fail ->
  logger.error("Object [{}] failed to transfer", fail.toString()));

}

Possible Solution

No response

Additional Information/Context

My pom.xml includes:

software.amazon.awssdk bom 2.20.79 pom import software.amazon.awssdk.crt aws-crt 0.22.1 software.amazon.awssdk s3-transfer-manager

AWS Java SDK version used

2.20.79

JDK version used

openjdk version "1.8.0_352-beta"

Operating System and version

Windows 10 Enterprise

The text was updated successfully, but these errors were encountered:

debora-ito · 2023-07-05T19:15:40Z

DownloadDirectoryRequest downloadDirectoryRequest = DownloadDirectoryRequest.builder()
  .destination(Paths.get(destinationPath))
  .bucket(this.s3BucketName + "/" + prefix)  //don't append the prefix here
  .build();

@rickyeng127 the reason you're seeing a signature error is because the prefix is appended to the end of the bucket name. S3 is expecting to get a bucket name but is getting a bucket+prefix instead.

Try providing the bucket name alone, and provide a prefix in the listObjectsRequestTransformer:

DownloadDirectoryRequest downloadDirectoryRequest = DownloadDirectoryRequest.builder()
    .destination(Paths.get(PATH))
    .bucket(BUCKET_NAME)
    .listObjectsV2RequestTransformer(transformer -> transformer.prefix(PREFIX))
    .build();

rickyeng127 · 2023-07-05T22:38:15Z

Hi @debora-ito , When I updated my code to use listObjectsV2RequestTransformer, I am seeing a different exception:

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send request

My code is modified as follows:

public void downloadFilesToDirectory(String destinationPath, String prefix) throws InterruptedException {

S3AsyncClient s3AsyncClient =
    S3AsyncClient.crtBuilder()
      .targetThroughputInGbps(20.0)
      .minimumPartSizeInBytes(1000000L)
      .build();

S3TransferManager transferManager = S3TransferManager.builder()
    .s3Client(s3AsyncClient)
    .build();

DownloadDirectoryRequest downloadDirectoryRequest = DownloadDirectoryRequest.builder()
  .destination(Paths.get(destinationPath))
  .bucket(this.s3BucketName)
  .listObjectsV2RequestTransformer(transformer -> transformer.prefix(prefix))
  .build();

DirectoryDownload directoryDownload = transferManager.downloadDirectory(downloadDirectoryRequest);

CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

completedDirectoryDownload.failedTransfers().forEach(fail ->
  logger.error("Object [{}] failed to transfer", fail.toString()));

}

Is there any trace logging that I can enable to troubleshoot this?

github-actions · 2023-07-07T16:02:16Z

COMMENT VISIBILITY WARNING

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

rickyeng127 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 28, 2023

debora-ito added closing-soon This issue will close in 2 days unless further comments are made. and removed needs-triage This issue or PR still needs to be triaged. labels Jul 5, 2023

debora-ito self-assigned this Jul 5, 2023

github-actions bot removed the closing-soon This issue will close in 2 days unless further comments are made. label Jul 6, 2023

rickyeng127 closed this as completed Jul 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Downloading Directory with SSE KMS encryption from S3 bucket fails #2995

Downloading Directory with SSE KMS encryption from S3 bucket fails #2995

rickyeng127 commented Jun 28, 2023

debora-ito commented Jul 5, 2023

rickyeng127 commented Jul 5, 2023 •

edited

github-actions bot commented Jul 7, 2023

Downloading Directory with SSE KMS encryption from S3 bucket fails #2995

Downloading Directory with SSE KMS encryption from S3 bucket fails #2995

Comments

rickyeng127 commented Jun 28, 2023

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Java SDK version used

JDK version used

Operating System and version

debora-ito commented Jul 5, 2023

rickyeng127 commented Jul 5, 2023 • edited

github-actions bot commented Jul 7, 2023

COMMENT VISIBILITY WARNING

rickyeng127 commented Jul 5, 2023 •

edited