Some issues with the AWS S3 assetstore

[fepegar]

Hi and thanks for this great project that we're planning to use soon. I would like to have support for Azure at some point (#3375), but first I'm trying to make AWS S3 work. I have some questions, issues and comments.

CORS configuration

In the Assetstores docs, there is a suggested CORS configuration:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <ExposeHeader>ETag</ExposeHeader>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

When I tried to add that XML to the CORS config on the AWS console, I was asked for a JSON instead. After some tweaks, comparing to some example provided in their docs and debugging messages from their API, I think the following is equivalent:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "PUT",
            "POST"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "ETag"
        ],
        "MaxAgeSeconds": 3000
    }
]

Should that be changed in the docs? I'm happy to submit a PR, if you like.

API error

I managed to mount the bucket, or at least I can see the files in the folder I imported it on. However, the files can't be downloaded. When I click on a TIFF slide, I'm redirected to an XML showing the following:

<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing
        method.</Message>
    <AWSAccessKeyId>AKIA3MNTRYHZCITTLUWA</AWSAccessKeyId>
    <StringToSign>AWS4-HMAC-SHA256 20220226T133542Z 20220226/eu-west-3/s3/aws4_request
        30bddb6d93d5e5a15068be7ebd3c0544ba3af6b6560706cd79d6bc2615e77bf0</StringToSign>
    <SignatureProvided>1faca4a0c6b60d08eadfb068bd34c8334bb3359be85ebab2a374b05567deb5ff</SignatureProvided>
    <StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 32 30 32 32 36 54 31 33 33 35 34 32
        5a 0a 32 30 32 32 30 32 32 36 2f 65 75 2d 77 65 73 74 2d 33 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a
        33 30 62 64 64 62 36 64 39 33 64 35 65 35 61 31 35 30 36 38 62 65 37 65 62 64 33 63 30 35 34 34 62 61 33 61 66
        36 62 36 35 36 30 37 30 36 63 64 37 39 64 36 62 63 32 36 31 35 65 37 37 62 66 30</StringToSignBytes>
    <CanonicalRequest>GET /192863a82b5a954ba0fa56b910574e1a.tiff
        X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA3MNTRYHZCITTLUWA%2F20220226%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Date=20220226T133542Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host
        host:dsas3.s3.eu-west-3.amazonaws.com host UNSIGNED-PAYLOAD</CanonicalRequest>
    <CanonicalRequestBytes>47 45 54 0a 2f 31 39 32 38 36 33 61 38 32 62 35 61 39 35 34 62 61 30 66 61 35 36 62 39 31 30
        35 37 34 65 31 61 2e 74 69 66 66 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43
        2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 33 4d 4e 54 52 59 48 5a
        43 49 54 54 4c 55 57 41 25 32 46 32 30 32 32 30 32 32 36 25 32 46 65 75 2d 77 65 73 74 2d 33 25 32 46 73 33 25
        32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 32 30 32 32 36 54 31 33
        33 35 34 32 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64
        48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 64 73 61 73 33 2e 73 33 2e 65 75 2d 77 65 73 74 2d 33 2e
        61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44
    </CanonicalRequestBytes>
    <RequestId>Z3RJYD5V6C183RH5</RequestId>
    <HostId>NmNcFlc7L3ad27UQM/2eVo1QDCM4fDzaoW0nYKDQqwhOVjko/OPwMGsFsI3ttP2NH0lVzI135o8=</HostId>
</Error>

I created an access key for myself at https://console.aws.amazon.com/iam/home#/security_credentials, and I've doucble-checked that they are correct in the assetstore config. Do you know what could be wrong?

Please let me know if you need any further info.

[zachmullen]

Should that be changed in the docs? I'm happy to submit a PR, if you like.

Yes please, if the XML has been deprecated or removed, we should certainly update the docs.

Regarding the signature error, I am wondering if the region is configured correctly on your assetstore? Is it set to eu-west-3?

[fepegar]

Yes please, if the XML has been deprecated or removed, we should certainly update the docs.

Yes, it looks like it has been changed. I'll open the PR in a minute.

Regarding the signature error, I am wondering if the region is configured correctly on your assetstore? Is it set to eu-west-3?

Yes. I wasn't sure how to fill that in, by the way. Maybe the docs could explain how to get the region code from the AWS console.

[fepegar]

I'll open the PR in a minute.

Actually, should I wait until my issue is fixed, just to confirm that it's not related to the CORS config?

[zachmullen]

I don't suspect it's related to the CORS config. I am going to try and dig in a little more on this and see if others have encountered this.

[zachmullen]

I was just checking the S3 docs and it seems like XML is still supported. What issue were you seeing when trying with the XML?

[fepegar]

I was just checking the S3 docs and it seems like XML is still supported. What issue were you seeing when trying with the XML?

Unless I'm in the wrong place, the console seems to want a JSON:

[zachmullen]

Hm, it does say that, but what happens if you do put the XML in and save it?

[fepegar]

I tried, it complained.

[fepegar]

More specifically, this error message is

The CORS configuration must be written in valid JSON.

API response
Expected params.CORSConfiguration.CORSRules to be an Array

[zachmullen]

Oh interesting, I guess their own docs are wrong (and it's far from the first time). Ok, let me dig into it further.

[fepegar]

Update

I have now deployed the DSA on a virtual machine, instead of a multi-container application. That allows me to perform OAuth authentication (#3382) and I'm not getting the error I reported above (#3379 (comment)) but the thumbnails are not being shown, and when I click on an image, the browser tries to download it (instead of opening it).

[manthey]

If you aren't using the docker file with DSA, then you probably aren't running "girder mount" on the same machine as Girder. Some image sources require actual files (not just Python file-like objects) to read the image, and "girder mount" provides a FUSE file system that maps any assetstore (include S3) to a file system so that libraries that require files work.

[manthey]

If switching from containerization to a virtual machine fixed an issue, it sounds like some port that was necessary wasn't open.

[fepegar]

If you aren't using the docker file with DSA, then you probably aren't running "girder mount" on the same machine as Girder.

I've been using the docker-compose file in minimal. I have just tried deploying the full thing with deploy.sh and have managed to get access to the files on AWS normally. If you like, I can confirm that it's not working with the minimal deployment. Otherwise, I'm happy to close this issue.

[manthey]

Yes -- the minimal example doesn't mount fuse. Probably we should update the docs on the minimal example to be more explicit that it expects all the data to be local to it for it to work.

[fepegar]

Thanks, that was very helpful! Running with deploy.sh, I can also see mounted folders now.