Support additional Amazon S3 checksum algorithms for @uppy/aws-s3-multipart and @uppy/companion

[daghendrik]

Amazon S3’s PutObject function now lets you specify the use of any one of four widely used checksum algorithms (SHA-1, SHA-256, CRC-32, and CRC-32C) when you upload each of your objects to S3, in addition to MD5. (More details on Amazon's blog post from February 2022)

It would be great if Uppy could support this for S3 Multipart uploads, letting us choose from other algorithms than just MD5, depending on our needs (speed, security, etc.).

A gist of how this works has been created by @hguillermo (comment) in issue #3391

What I believe needs to be changed in the codebase is:

Companion

1. The params in function createMultipartUpload in uppy/companion/src/server/controllers/s3.js would have to allow for ChecksumAlgorithm to be set ( values: "CRC32"|"CRC32C"|"SHA1"|"SHA256"|string; )
2. The functions handling the signing of the parts in s3.js (signPartUpload and batchSignPartsUpload) would have add support for the keys ChecksumCRC32, ChecksumCRC32C, ChecksumSHA1, ChecksumSHA256 in the object passed to s3Client. The checksum values would then be sent in as a parameter from the Uppy Client.

Uppy Client

On the client side the browser would have to calculate a the checksum (CRC32, CRC32C, SHA1 or SHA256) of each individual chunk (just like this suggestion for calculateMD5), pass it to the companion, and once a signed url is received (from Companion or a custom backend) I believe you also have do add the chunk's checksum value as a header sent along the S3 PUT ("x-amz-checksum-crc32", "x-amz-checksum-crc32c", etc)

• https://docs.amazonaws.cn/en_us/AmazonS3/latest/API/API_PutObject.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
• https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/

Another requirement for this to be possible is that individual headers can be sent along each chunk, which is already addressed in the case for being able to verify the integrity using the "Content-MD5" header. See issue #3881 by @kevin-west-10x

[Murderlon]

Thanks for the detailed issue. This helps a lot.

[daghendrik]

@Murderlon ,
Thanks for adding the possibility to send unique headers along with each chunk in @uppy/aws-s3-multipart in Uppy 3.x (#3895).

I haven't upgraded my web app to Uppy 3 yet, but from first glance it seems that I can use the new S3 Multipart custom header support to pass along other checksums than "Content-MD5" to S3 for each upload-part, for instance"x-amz-checksum-crc32", "x-amz-checksum-crc32c", etc.

Since I'm using a custom backend to recreate the companion API for the Uppy client, I'll be able to set ChecksumAlgorithm in my backend code to my desired value ("CRC32"|"CRC32C"|"SHA1"|"SHA256"|string; ). I'll try to find time to experiment with this next week. However, it would be even better if this was something that Companion supported natively. From what I can see, it would require these changes:

1. The params in function createMultipartUpload in uppy/companion/src/server/controllers/s3.js would have to allow for ChecksumAlgorithm to be set ( values: "CRC32"|"CRC32C"|"SHA1"|"SHA256"|string; )
2. The functions handling the signing of the parts in s3.js (signPartUpload and batchSignPartsUpload) would have to add support for the keys ChecksumCRC32, ChecksumCRC32C, ChecksumSHA1, ChecksumSHA256 in the object passed to s3Client.
3. The Uppy client functions for createMultiPartUpload and prepareUploadParts would have to add optional parameters for checksum algorithm.

Do you think there's a possibility of this being implemented in @uppy/companion & @uppy/aws-s3-multipart anytime soon?

[Murderlon]

Hi, thanks for the update. We currently have two focus months with some priorities (not just Uppy, also other orgs like Tus) and @uppy/aws-s3-multipart is not part of that. So if I'm totally honest I don't think we'll add this soon. If you feel this should be added sooner, we can help along if you create a PR for it.

[daghendrik]

Thanks for the quick and honest reply!