@uppy/aws-s3-multipart Replace functionality lost in v2 by batching with prepareUploadParts

[kevin-west-10x]

The breaking change in v2 of @uppy/aws-s3-multipart that migrates prepareUploadPart to prepareUploadParts loses some fine grained functionality that in my case is preventing me from updating from v1. I need this functionality in order to calculate md5 hashes of the parts I'm preparing so they can be validated on s3. The two changes I can see that caused an issue are:

1. No longer passing file data of the part(s) we are preparing. This was fixed in Add chunks back to prepareUploadParts, indexed by partNumber #3520 with help from @Murderlon.
2. The headers passed to s3 in the return payload are now shared among all of the batched parts. Due to this I can no longer set the Content-MD5 of each individual part.

Being able to batch create the presigned urls isn't a bad option to have, but it would be nice if this functionality could be restored in some way.

[Murderlon]

Current return value of prepareUploadParts:

{
  "presignedUrls": {
    "1": "https://bucket.region.amazonaws.com/path/to/file.jpg?partNumber=1&...",
    "2": "https://bucket.region.amazonaws.com/path/to/file.jpg?partNumber=2&...",
    "3": "https://bucket.region.amazonaws.com/path/to/file.jpg?partNumber=3&..."
  },
  "headers": { "some-header": "value" }
}

Proposal:

function prepareUploadParts(file, { uploadId, key, parts }) {
  const presignedRequests = parts.map((part) => {
    const { number, chunk } = part;
    const url = `https://bucket.s3.us-east-2.amazonaws.com/test/upload/multitest.dat?partNumber=${number}&...`;
    const hash = calculateMD5(chunk);
    return new Request(url, { headers: { "Content-MD5": hash } });
  });

  return { presignedRequests };
}

Changes:

• partNumbers and chunks have now been merged into parts (Array<{ number: number, chunk: blob }>). I think they were separate to prevent breaking changes, but they belong together.
• Return presignedRequests as an array with Request's.
  • It felt a bit off to make it "1": { url, headers }. Also making headers part indexed felt a bit off, but possible I suppose.
  • This way we 'use the platform'
  • Downside is you can set things in Request like the method and perhaps some other things that shouldn't happen. We could leave it up to the implementer to not mess that up, or always override those things.

---

Curious about your thoughts @mifi @aduh95 @arturi @martin-brennan

[daghendrik]

Thanks for moving this forward, @Murderlon and @kevin-west-10x . It will be very useful for us as well.

If possible, it would be awesome to also support the other/additional checksums that AWS now accepts as integrity checks on S3 (https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/)

[Murderlon]

If possible, it would be awesome to also support the other/additional checksums that AWS now accepts as integrity checks on S3 (https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/)

I'm not immediately sure what it takes for us to support those. Probably better to tackle as a separate issue. Would you be willing to investigate a bit to create an issue with pointers what needs to change in the codebase?

[daghendrik]

@Murderlon : Sure, added an issue: #3885

[mifi]

sounds good, but should file be part of the object arg? and does it need to return an object?

one concern is maybe some users need to do asynchronous calculations in order to get the hash. so maybe the function should be async?

what about this:

async function prepareUploadParts({ file, uploadId, key, parts }) {
  return parts.map((part) => {
    const { number, chunk } = part;
    const url = `https://bucket.s3.us-east-2.amazonaws.com/test/upload/multitest.dat?partNumber=${number}&...`;
    const hash = calculateMD5(chunk);
    return new Request(url, { headers: { "Content-MD5": hash } });
  });
}

[Murderlon]

sounds good, but should file be part of the object arg? and does it need to return an object?

We could do that too, yes.

one concern is maybe some users need to do asynchronous calculations in order to get the hash. so maybe the function should be async?

prepareUploadParts is already async and will stay that way.

[Murderlon]

Closed in #3895