Does form-data support fs-capacitor stream?

[igat64]

I'm using apollo-upload-server for uploading files to the server. In my resolver function, I have a stream. What I want to do is pass this stream as a file (multipart/form-data) to another server. I use promise-request (that under the hood use form-data to achieve that. Code look like below:

import * as request from 'request-promise';

async upload(_, { file }) {
  const { stream } = await file;
  return await request.post({
      uri: `${NLP_API_BASE}/upload`,
      formData: {
        datasheet: stream
      }
    });
}

But I'm getting an error :
RequestError: Error: socket hang up at new RequestError (/Users/igat/Code/projects/botsupply-oracle-backend/node_modules/request-promise-core/lib/errors.js:14:15) at Request.plumbing.callback (/Users/igat/Code/projects/botsupply-oracle-backend/node_modules/request-promise-core/lib/plumbing.js:87:29) at Request.RP$callback [as _callback] (/Users/igat/Code/projects/botsupply-oracle-backend/node_modules/request-promise-core/lib/plumbing.js:46:31) at self.callback (/Users/igat/Code/projects/botsupply-oracle-backend/node_modules/request/request.js:185:22) at emitOne (events.js:121:20) at Request.emit (events.js:211:7) at Request.onRequestError (/Users/igat/Code/projects/botsupply-oracle-backend/node_modules/request/request.js:877:8) at emitOne (events.js:116:13) at ClientRequest.emit (events.js:211:7) at Socket.socketOnEnd (_http_client.js:423:9) at emitNone (events.js:111:20) at Socket.emit (events.js:208:7) at endReadableNT (_stream_readable.js:1056:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9)

And as I understand apollo-upload-server use fs-capacitor to deal with a stream. Does form-data support fs-capacitor stream? What is wrong with my code?

Guys any ideas? I'm really stuck.

[rheaditi]

Have a similar issue too! :(

Did anyone find a solution? It's been over a year and I'm still unable to find many references to an issue like this.

[amannn]

@rheaditi I tested a few different approaches and ended up with this resolver for a file upload with apollo-upload-server@7.0.0.

import FormData from 'form-data';
import {GraphQLUpload} from 'apollo-upload-server';
import rawBody from 'raw-body';
import fetch from 'node-fetch';

export default {
  Upload: GraphQLUpload,

  Mutation: {
    uploadFile: (root, {file: fileUpload}) =>
      fileUpload.then(file =>
        rawBody(file.createReadStream()).then(buffer => {
          const form = new FormData();

          // `FormData` accepts either a readable stream, a buffer or a string
          // for multipart files. `file.createReadStream()` creates a
          // `FileStream`, which is a subclass of `ReadableStream` implemented
          // in `busboy` (dependency of `apollo-upload-server`). This kind of
          // stream is apparently not discovered correctly by `form-data`,
          // therefore the better option is to use a buffer, however the meta
          // data needs to be specified explicitly in this case.
          form.append('file', buffer, {
            filename: file.filename,
            contentType: file.mimetype,
            knownLength: buffer.length
          });

          // Something like this, I'm using some abstraction for this
          return fetch('/upload', {method: 'POST', body: form});
        })
      )
  }
};

Maybe this is helpful to you! I haven't tested with newer versions of apollo-upload-server yet.

[rheaditi]

Hey @amannn!
Just tried your approach and it seems to work!

To get around this, we previously had to store the file to the fs temporarily & create a readstream from the fs file and this helps us completely avoid that! You, sir, are a life-saver.

Thank you so much!!! 🙏

---

EDIT:
Adding the versions for anyone checking this:

├─┬ apollo-datasource-rest@0.6.4
│ └─┬ apollo-server-env@2.4.3
│   └── node-fetch@2.6.0
├─┬ apollo-server@2.9.4
│ ├─┬ apollo-server-core@2.9.4
│ │ ├─┬ @apollographql/apollo-tools@0.4.0
│ │ │ └─┬ apollo-env@0.5.1
│ │ │   └── node-fetch@2.6.0  deduped
│ │ └── graphql-upload@8.0.7
│ └── apollo-server-express@2.9.4  deduped
├─┬ apollo-server-express@2.9.4
│ └─┬ body-parser@1.19.0
│   └── raw-body@2.4.0
├── form-data@2.5.1
└── raw-body@2.4.1

❯ node -v
v12.8.1

[amannn]

To get around this, we previously had to store the file to the fs temporarily & create a readstream from the fs file and this helps us completely avoid that!

I've been there before as well 😅.

Glad I could help you! 🙂

[abinavseelan]

To get around this, we previously had to store the file to the fs temporarily & create a readstream from the fs file and this helps us completely avoid that!

We'd been stuck with this too. 😅

Thank you so much for this @amannn 🙌

[gregory]

Isn't rawBody(file.createReadStream()) going to load the all file into memory though? ...

[amannn]

@gregory Yep, that's true. Ideally we'd just forward the stream to form-data, but as mentioned above in my code comment, there's a bug that prevents this currently. My solution is just a workaround for the time being.

[gregory]

@gregory Yep, that's true. Ideally we'd just forward the stream to form-data, but as mentioned above in my code comment, there's a bug that prevents this currently. My solution is just a workaround for the time being.

Got it @amannn - have a look at jaydenseric/graphql-upload#168 - I created a new option called waitForFileOnDisk which when set to true, will wait for the file to be written on disk before resolving.

[mike-marcacci]

Hi everybody. I want to clear up a few things here. The problem is that the form-data library is looking for a .path property on the readable stream, and using it to make two kinds of inferences:

1. The file size:
   

   form-data/lib/form_data.js

   Lines 135 to 147 in 5c9b3a9

   	fs.stat(value.path, function(err, stat) {
   	var fileSize;
   	if (err) {
   	callback(err);
   	return;
   	}
   	// update final size based on the range options
   	fileSize = stat.size - (value.start ? value.start : 0);
   	callback(null, fileSize);
   	});

2. The file type:
   

   form-data/lib/form_data.js

   Lines 254 to 256 in 5c9b3a9

   	if (!contentType && value.path) {
   	contentType = mime.lookup(value.path);
   	}

The fs-capacitor library is a general purpose streaming library that uses the filesystem under the hood, but in ways that invalidate both of these inferences, and in fact its .path property is simply an undocumented internal implementation detail. In the most recent versions, I've renamed it to ._path to make this more clear.

To resolve this, form-data just needs to add support for arbitrary stream data, instead of using duck typing to only support the special cases of the fs and http libraries.

[gregory]

You got it right @mike-marcacci that's the issue.
That's why I exposed a new option in jaydenseric/graphql-upload#168 , to allow any other library relying on the file size & file type, to work with graphql-upload.

Instead of trying to refactor form-data and going down a rabbit hole, I thought enhancing graphql-upload through a two line change was smarter.

If you are resistant to that small change, it looks like the only quick fix is do what @amannn suggested, but at the cost of loading every single file upload in memory.

Let me know if you think this was a stupid idea.

[mike-marcacci]

Hi @gregory, one of the problems with using the undocumented .path property (now removed in more recent versions) on fs-capacitor streams is that you will get the wrong values for name, type, and size, since:

• the filename used is a randomly generated string
• the file size is not static (the file itself can still be growing, with fs-capacitor coordinating read and write streams)

There isn't a way for form-data to correctly use this field even if we made sure it was available; it just happened to be named the same thing. Hopefully that makes sense!

[gregory]

I don't think that's an issue @mike-marcacci as you'll usually set those manually:

          // data needs to be specified explicitly in this case.
          form.append('file', buffer, {
            filename: file.filename,
            contentType: file.mimetype,
          });

As you can see, those will be returned by graphql-upload

[yaacovCR]

Figured it out -- it's not an issue of duck typing per se, see node-fetch/node-fetch#707.

FormData just does not purposefully support streams of unknown length, unless you use NaN workaround.

[yaacovCR]

graphql-tools-fork v8.1.0 provides new exports to allow proxying remote file uploads for those who want to proxy all the things.

It successfully sends a server-to-server multipart form request via FormData and node-fetch by extending the FormData class to handle streams of unknowable length.

See: https://github.com/yaacovCR/graphql-tools-fork/blob/master/src/links/createServerHttpLink.ts#L15-L63

Feedback/suggestions welcome. I am happy to submit as a PR to this repository if there would be support for having it merged.

[lynxtaa]

It works for me with additional headers (using node-fetch)

const { filename, createReadStream } = await file

const form = new FormData()

form.append('file', createReadStream(), { filename })

await fetch(FILE_SERVER_URL, {
  method: 'PUT',
  body: form,
  headers: {
    'connection': 'keep-alive',
    'transfer-encoding': 'chunked',
  },
})

[angusfretwell]

I'm having a bit of a hard time following this issue. What's the status of it, and is there a reasonable workaround (i.e. one that doesn't involve loading the entire fire into memory)?

[yaacovCR]

@lynxtaa, that doesn't work for me. When using those headers from Node v12 on Windows to express server on same, I get the following error:

request to http://localhost:50138/ failed, reason: read ECONNRESET

My summary of the issue is that the combination of the existing Node FormData and fetch polyfills do not allow adding arbitrary streams, as unless they are specific streams, the content length will be set improperly.

Available Workarounds:

1. Specifically set contentLength as NaN, and use node-fetch v3 (release?) that has been been patched to not include content length if the length comes back as NaN.
2. Use patched FormData from Does form-data support fs-capacitor stream? #394 (comment) or something like it. This could be submitted as PR to FormData, still waiting for any interested from this package's maintainers.
3. Copy the file into memory.
4. Depending on server?, adding headers as per @lynxtaa's suggestion.
5. ?

[drzraf]

I can confirm uploading using FormData + node-fetch is currently broken on node 10.
Workaround: use form.append('foo', fs.readFileSync(filename));

[lynxtaa]

@lynxtaa, that doesn't work for me. When using those headers from Node v12 on Windows to express server on same, I get the following error:

request to http://localhost:50138/ failed, reason: read ECONNRESET

My summary of the issue is that the combination of the existing Node FormData and fetch polyfills do not allow adding arbitrary streams, as unless they are specific streams, the content length will be set improperly.

Available Workarounds:

1. Specifically set contentLength as NaN, and use node-fetch v3 (release?) that has been been patched to not include content length if the length comes back as NaN.

2. Use patched FormData from [#394 (comment)](https://github.com/form-data/form-data/issues/394#issuecomment-569850654) or something like it. This could be submitted as PR to FormData, still waiting for any interested from this package's maintainers.

3. Copy the file into memory.

4. Depending on server?, adding headers as per @lynxtaa's suggestion.

5. ?

After further investigation I think it works for me only because I'm using Nginx reverse proxy in front of my file-upload server. It somehow solves the issue. Maybe it has something to do with proxy_request_buffering? By default Nginx buffers a client request body

[nfadili]

@lynxtaa, that doesn't work for me. When using those headers from Node v12 on Windows to express server on same, I get the following error:

request to http://localhost:50138/ failed, reason: read ECONNRESET

My summary of the issue is that the combination of the existing Node FormData and fetch polyfills do not allow adding arbitrary streams, as unless they are specific streams, the content length will be set improperly.

Available Workarounds:

1. Specifically set contentLength as NaN, and use node-fetch v3 (release?) that has been been patched to not include content length if the length comes back as NaN.

2. Use patched FormData from [#394 (comment)](https://github.com/form-data/form-data/issues/394#issuecomment-569850654) or something like it. This could be submitted as PR to FormData, still waiting for any interested from this package's maintainers.

3. Copy the file into memory.

4. Depending on server?, adding headers as per @lynxtaa's suggestion.

5. ?

@yaacovCR Appreciate the summary. I have been struggling with this for awhile :D
Regarding (1) I am not able to recreate a success with this. I'm assuming by 'contentLength' you mean 'knownLength' in the form-data package? i.e.

const form: FormData = new FormData();
form.append('', readStream, { knownLength: NaN });
                    
const options = {
    method: 'POST',
    body: form,
    headers: {
        Authorization: `Bearer <token>`,
    }
};
await fetch(SERVER_URL, options)

The only way I can send a multipart/form request to the server I am dealing with (Power BI API) is by loading the entire file into memory or the filesystem and sending it's length along with the request. The server response always says the file size is invalid and says it's 9223372036854775807 bytes (max value of a int64) which makes me think there is some funky C# stuff going on as well... regardless, thanks for the ideas and I'll keep this thread in mind if I happen to come across any other solutions.

[hokaccha]

My workaround is using formdata-node.

import FormData from "formdata-node";

...

const form = new FormData();
form.set("file", file.createReadStream(), file.filename);

await fetch(url, {
  method: "POST",
  body: form.stream,
  headers: form.headers,
});

[crickford]

This issue appears to have been fixed by #382, released in v4.0.0