Improve error messages for S3 HeadObject

[mlafeldt]

While adding logging context to one of my projects, I noticed that both put_object and head_object only return a generic "Error" when lacking permissions (403). I would expect to see at least the string "forbidden" somewhere in the error message.

Here's the complete anyhow error chain from CloudWatch:

context: "failed to get metadata for object strips/2021-09-22.gif",
--
  | source: ServiceError {
  | err: HeadObjectError {
  | kind: Unhandled(
  | Error {
  | code: None,
  | message: None,
  | request_id: None,
  | extras: {},
  | },
  | ),
  | meta: Error {
  | code: None,
  | message: None,
  | request_id: None,
  | extras: {},
  | },
  | },
  | raw: Response {
  | inner: Response {
  | status: 403,
  | version: HTTP/1.1,
  | headers: {
  | "x-amz-request-id": "8SCE568PZ3EF7WVA",
  | "x-amz-id-2": "s4nNU4EWiGhx2vSzTOPBAhmHJRNpMYp6weyKP0QTjIFXXvVe4JyTLjAiTIZ6bWXwhlQ+pBwbC+E=",
  | "content-type": "application/xml",
  | "date": "Wed, 22 Sep 2021 20:41:00 GMT",
  | "server": "AmazonS3",
  | },
  | body: SdkBody {
  | inner: Once(
  | Some(
  | b"",
  | ),
  | ),
  | retryable: true,
  | },
  | },
  | properties: SharedPropertyBag(
  | Mutex {
  | data: PropertyBag,
  | poisoned: false,
  | ..
  | },
  | ),
  | },
  | },
  | }

Returning the error from Lambda currently results in:

{
  "errorType": "&anyhow::Error",
  "errorMessage": "Error"
}

[rcoh]

We could potentially generate an error message from the status code if no other message, but this is a bit risky because status codes aren't always used correctly.

Unfortunately, in this particular case, S3 isn't actually sending back anything other than a 403 anywhere in the response, so there's nothing the SDK can do here from the data we have.

[mlafeldt]

It would be great if the SDK could last least return AccessDenied for 403 and other common codes, see https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

Looking at the output of aws s3 ls --no-sign-request some-existing-bucket --debug, you can see that the response payload definitely contains "AccessDenied":

2021-09-28 09:19:28,503 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>...</RequestId><HostId>...</HostId></Error>'

But I guess the string isn't used by the Rust SDK right now?

Error output of the CLI:

aws s3 ls --no-sign-request some-existing-bucket

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

I'm also pretty sure that the Go SDK returns "AccessDenied" too.

[rcoh]

The problem seems to be exclusive to HeadObject: Since HeadObject can't return a body, so there's no error message in the body for us to parse and it isn't present in a header either:

http_response=Response { 
  status: 403, 
  version: HTTP/1.1, 
  headers: {"x-amz-request-id": "Z6JP0K096M149FP8", "x-amz-id-2": "nzyDVpG7PZzQwsDpH34j5U2rXISc0tS4BHDd3u6C+P6zfApqgrteKuvuIdB0smCaSiJ+akDAphE=", "content-type": "application/xml", "date": "Tue, 28 Sep 2021 12:57:58 GMT", "server": "AmazonS3"}, 
  body: b"" 
}

When I run put_object, I get a proper error with AccessDenied:

Error { code: Some("AccessDenied"), message: Some("Access Denied")

Can you verify you see that as well? If not, I wonder if the anyhow chain is dropping the error.

aws s3 ls is list_bucket I believe. The CLI is smart enough to turn 403 into Forbidden in the head object case and perhaps we should add that same logic as well.

➜  scratch git:(main) ✗ aws s3api head-object --bucket some-existing-bucket --key foo

An error occurred (403) when calling the HeadObject operation: Forbidden

[rcoh]

note for maintainers:
Since S3 doesn't actually model 403/Access denied (https://github.com/awslabs/aws-sdk-rust/blob/main/sdk/s3/src/error_meta.rs#L5-L13) we can't use the same customization pattern we use for 404 responses.

[mlafeldt]

You're absolutely right! I just found it in the CLI docs as well:

A HEAD request has the same options as a GET action on an object. The response is identical to the GET response except that there is no response body. Because of this, if the HEAD request generates an error, it returns a generic 404 Not Found or 403 Forbidden code. It is not possible to retrieve the exact exception beyond these error codes.

I also confirmed that it's not a problem with other API calls like PutObject:

{
  "errorType": "&anyhow::Error",
  "errorMessage": "Error { code: \"AccessDenied\", message: \"Access Denied\", request_id: \"7TSDETA3JTRTP0HJ\", s3_extended_request_id: \"qv9IfbAUAZ+/ZIvqbyjYCh9NDlNvuGL1Ws8hEBwiD7e0LnIVW/aFFUG1jFTTlNKFxlhhIL5h6uY=\" }"
}

The CLI is smart enough to turn 403 into Forbidden in the head object case and perhaps we should add that same logic as well.

Agreed. 👍🏻

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.