aws-s3-multipsart upload issue #3021

SamGoody · 2021-07-16T10:34:30Z

Thank you all for this amazing project.

I am trying to setup upload to AWS S3.
Regular uploads work, multipart does not.

Here is my test code (minimized to find issues).

index.js

const express = require('express')
const bodyParser = require('body-parser')
const session = require('express-session')
const companion = require('@uppy/companion')
const app = express()

app.use(bodyParser.json())
app.use(session({secret: ' secret'}))

const options = {
	  secret: ' secret',
	  providerOptions: {
		  s3: {
			key: "MYKEY",
			secret: "my+secret",
			bucket: "mybucket",
			region: "eu-west-2",
		    }
		},
	  server: {
		host: '68.183.149.146:3020',
		protocol: 'http',
		path: '/files',
	  },
	  filePath: '/var/www/uppy/uploads',
	}

app.use('/files', companion.app(options))
app.use(express.static('public'))

const server = app.listen(3020)
companion.socket(server, options)

public/index.html

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Uppy</title>
    <link href="https://releases.transloadit.com/uppy/v1.30.0/uppy.min.css" rel="stylesheet">
  </head>
  <body>
    <div id="drag-drop-area"></div>

    <script src="https://releases.transloadit.com/uppy/v1.30.0/uppy.min.js"></script>
    <script>
	const S3 = Uppy.AwsS3Multipart
        const uppy = Uppy.Core()
	        .use(Uppy.Dashboard, {
			inline: true,
			target: '#drag-drop-area'
			})
		.use(S3, {
			companionUrl: 'http://68.183.149.146:3020/files/'
			})
    </script>
  </body>
</html>

When I use const S3 = Uppy.AwsS3 everything works, the file uploads, and the sun shines.
When I use const S3 = Uppy.AwsS3Multipart I get an errors and it gets all dark and rainy.

Request URL: http://68.183.149.146:3020/files/s3/multipart
Request Method: POST
Status Code: 500 Internal Server Error
Remote Address: 68.183.149.146:3020
Referrer Policy: strict-origin-when-cross-origin

Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://68.183.149.146:3020
Access-Control-Expose-Headers: Access-Control-Allow-Headers
Connection: keep-alive
Content-Length: 64
Content-Type: application/json; charset=utf-8
Date: Fri, 16 Jul 2021 10:29:22 GMT
ETag: W/"40-Fy22roXZKeo8KYREwLLmWGRS9YI"
Keep-Alive: timeout=5
Set-Cookie: connect.sid=s%3AbtoYXU2S3b6eZ9KiPvjZOdIjTeTqJUMd.yOm4y6xNocW9R3l6oU8j0RPXfJprlOGZ0uZbWWzniJ0; Path=/; HttpOnly
Vary: Origin
X-Powered-By: Express

Request Payload
{filename: "image.png", type: "image/png", metadata: {name: "image.png", type: "image/png"}}

Response:
{"error":"s3: filename returned from `getKey` must be a string"}

(After some testing, it seems that (req, filename, metadata) is (express request object, undefined and {}).
According to the docs, it should be (undefined, image.png, {})
When I modify the getkey function to respond with a string, I get the following error.

Request URL http://68.183.149.146:3020/files/s3/multipart
Request Method: POST
Status Code: 400 Bad Request
Remote Address: 68.183.149.146:3020
Referrer Policy: strict-origin-when-cross-origin

Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://68.183.149.146:3020
Access-Control-Expose-Headers: Access-Control-Allow-Headers
Connection: keep-alive
Content-Length: 45
Content-Type: application/json; charset=utf-8
Date: Fri, 16 Jul 2021 09:34:49 GMT
ETag: W/"2d-mGfo8ml5NA1R+iB4e6ecoWPed+s"
Keep-Alive: timeout=5
Vary: Origin
X-Powered-By: Express

Request Payload:
{"filename":"image.png","type":"image/png","metadata":{"name":"image.png","type":"image/png"}} 

Response:
{"error":"s3: content type must be a string"}

On the S3 bucket, the CORS headers are:

[{
    "AllowedHeaders": [ "*" ],
    "AllowedMethods": [ "GET", "POST", "PUT" ],
    "AllowedOrigins": [ "*" ],
    "ExposeHeaders": [ "ETag" ],
    "MaxAgeSeconds": 3000
}]

But I assume that my bucket is setup correctly, because the AWS.S3 upload works.
Is there something else I need to change to get multipart?

The text was updated successfully, but these errors were encountered:

Murderlon · 2021-08-10T11:15:42Z

@mifi do you perhaps know what's going on here?

mifi · 2021-08-10T14:59:38Z

Nope,but I could look into it.

Add the following headers to access-control-allow-headers by default: Authorization, Origin, Content-Type, Accept They are needed for basic operation. See #3021 therefore also remove custom middlewares in examples and standalone

mifi · 2021-09-03T15:26:53Z

Thanks for providing sample code to reproduce. Because of that I have figured out what is the problem. The problem is that the example code you probably copied from here and based your code on did not contain the correct CORS headers (in this case content-type). This meant the client wouldn't sent content-type: application/json but instead text/plain, and so express couldn't parse the body and req.body.filename was null and you get 500.

What you can do to fix this now is to add the following before your app.use('/files', companion.app(options)):

app.use((req, res, next) => {
  res.setHeader(
    'Access-Control-Allow-Headers',
    'Authorization, Origin, Content-Type, Accept'
  )
  next()
})

That should fix it for you. This was a very confusing bug,so I will create a PR to make this header is added by default so this code will not be necessary.

* refactor cors middleware to avoid duplicates * Make more ACAH headers default Add the following headers to access-control-allow-headers by default: Authorization, Origin, Content-Type, Accept They are needed for basic operation. See #3021 therefore also remove custom middlewares in examples and standalone * Update outdated readme for S3 AWS now requires JSON instead of XML format * Update packages/@uppy/companion/src/server/middlewares.js Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com> * Update packages/@uppy/companion/src/server/middlewares.js Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com> * fix review comments Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

SamGoody · 2021-12-07T21:45:57Z

Thank you so much!

Debugging is a beast, and your solution works great.

* refactor cors middleware to avoid duplicates * Make more ACAH headers default Add the following headers to access-control-allow-headers by default: Authorization, Origin, Content-Type, Accept They are needed for basic operation. See transloadit#3021 therefore also remove custom middlewares in examples and standalone * Update outdated readme for S3 AWS now requires JSON instead of XML format * Update packages/@uppy/companion/src/server/middlewares.js Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com> * Update packages/@uppy/companion/src/server/middlewares.js Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com> * fix review comments Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

SamGoody added Bug Triage labels Jul 16, 2021

Murderlon added Question and removed Bug Triage labels Aug 10, 2021

mifi mentioned this issue Sep 3, 2021

Default allow headers #3167

Merged

Murderlon linked a pull request Sep 8, 2021 that will close this issue

Default allow headers #3167

Merged

mifi closed this as completed in #3167 Sep 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws-s3-multipsart upload issue #3021

aws-s3-multipsart upload issue #3021

SamGoody commented Jul 16, 2021 •

edited by aduh95

Murderlon commented Aug 10, 2021

mifi commented Aug 10, 2021

mifi commented Sep 3, 2021

SamGoody commented Dec 7, 2021

aws-s3-multipsart upload issue #3021

aws-s3-multipsart upload issue #3021

Comments

SamGoody commented Jul 16, 2021 • edited by aduh95

Murderlon commented Aug 10, 2021

mifi commented Aug 10, 2021

mifi commented Sep 3, 2021

SamGoody commented Dec 7, 2021

SamGoody commented Jul 16, 2021 •

edited by aduh95