Request timeout on multipart

[dscape]

Context

nano is getting a major overhaul, and I tried to update request since some of the nano userbase wanted some of the newest features.

However one of the multipart CouchDB tests broke. We simply never get a response from CouchDB.

Versions

CouchDB

$ curl 127.0.0.1:5984
{"couchdb":"Welcome","uuid":"baee9a6d05c9d82cde8244826d78f81f","version":"1.6.0","vendor":{"version":"1.6.0","name":"The Apache Software Foundation"}}

Node

$ node -e "console.log(process.versions)"
{ http_parser: '1.0',
  node: '0.10.26',
  v8: '3.14.5.9',
  ares: '1.9.0-DEV',
  uv: '0.10.25',
  zlib: '1.2.3',
  modules: '11',
  openssl: '1.0.1e' }

Request

$ cat node_modules/request/package.json | json version
2.47.0

How to reproduce

Assuming you have CouchDB installed and running on port 5984.

curl -vX PUT http://127.0.0.1:5984/mikeal

Or, if you use CouchDB frequently enough and want to have a CLI for it:

npm i futon -g
futon database create mikeal

The following timeouts, when I thought it should insert the multi-part attachment

var request = require('request');
var thisReq = request({ 
  method: 'PUT',
  headers: { 
    'content-type': 'multipart/related' 
  }, 
  uri: 'http://localhost:5984/mikeal/doc', 
  multipart: [ 
    { 
      'content-type': 'application/json', 
      body: '{"_attachments":{"att":{"follows":true,"length":12}},"foo":"baz"}' 

    },
    { body: 'Hello World!' }
  ] 
}, function (e, h, b) { console.log(e,b); });

However in 2.42.x, 2.45.x, and 2.46.x it logs:

> null '{"ok":true,"id":"doc","rev":"1-4bd1a5f94b0171823b5409250aeb5894"}\n'

[tbuchok]

hey nuno -- is this output the result of some logging on the request itself? is there a code sample using the request module that might help us all take a look together?

maybe i'm missing something.

[dscape]

@tbuchok sorry, you are right, this was lacking lots of context. I re-wrote this so you can chip in. Maybe even the person responsible for all the multipart changes on the last version?

[tbuchok]

@nylen i've been a little quiet lately and didn't follow most of the recent PRs -- is there anything that comes to mind / recent committers who the above context might spark a few ideas.

thanks, @dscape. i'll take a look through the tests as soon as i can and see if i can replicate -- you been able to replicate outside of couch-db connections?

[simov]

I made the changes, multipart/related uses combined-stream which is used by form-data as well. If that can provide any clue?

[tbuchok]

thanks @simov.

@dscape are you able to try this in v0.8 and see if what you're seeing here is the stream isn't changing to flowing mode -- and this is actually a Streams2 thingie? that's the very first thing that comes to mind...

if there's an easier way to test that out anyone have thoughts? in the middle of some other stuff at the moment -- sorry for the noise if this is off base.

[dscape]

I rolled back to 2.46.x for the time being.

Will try 0.8 soon

[simov]

It works for me on v0.8, as well as on my gdrive multipart test. That's my only real world test case ATM (unit tests are fine as usual). Apart from that the multipart/related implementation follows the one found in the form-data module.

But @dscape didn't used that either with couchdb, so it might be something specific to couchdb. I tried a few things and I'm getting different types of errors, so I can't say what it is yet.

[simov]

I just installed Couchdb1.6.1
with 2.46 this one uploads without a problem

request.put('http://localhost:5984/mikeal/cat', {
  multipart: [
    {
      'content-type': 'application/json',
      body: JSON.stringify({
        _attachments: {
          'cat.png': {
            follows: true,
            content_type: 'image/png',
            length: 22025
          }
        }
      })
    },
    {
      body: fs.readFileSync(image)
    }
  ]
}, function (err, res, body) {
  if (err) console.log(err);
  console.log(body);
});

with 2.47 the exact same code is constantly giving me

{"error":"bad_request","reason":"invalid_json"}

so I can't even get to a timeout bug yet.

This is the related PR #1185 and as you can see there are only two minor changes.

This is the Gdrive example, that works without a problem with 2.47

request.post('https://www.googleapis.com/upload/drive/v2/files', {
  qs:{
    access_token:'..',
    uploadType:'multipart'
  },
  multipart: [
    {
      'Content-Type':'application/json',
      body:JSON.stringify({
        title:'cat.png'
      })
    },
    {
      'Content-Type':'image/png',
      body:fs.createReadStream(image)
    }
  ]
},
function (err, res, body) {
  if (err) console.log(err);
});

[simov]

That's the cause of my invalid JSON using 2.47

Invalid JSON: {{error,garbage_after_value},
[<<"\r\n57\r\n{\"_attachments\":{\"cat.png\":{\"follows\":true,\"content_type\":\"image/png\",\"length\":22025}}}\r\n2\r\n\r\n\r\n2a">>]}

Some weird symbols, the JSON itself is correct. Any ideas?

[simov]

Wow, I was about to say that Couchdb have a long running record of not knowing how to parse multipart body with transfer-encoding:chunked set, and stumbled upon this - Connection hangs on document update for multipart/related and transfer encoding chunked request So it seems this should be fixed in their next release (again).

The question is, do we have to support the other variant of sending too (via option or by setting a header)? I'm not sure if any other server could have the same problems, but it's a breaking change nevertheless.

I'm for having the current implementation enabled by default. It's particularly useful for streaming large files to GDrive, also it unifies the experience with the formData option.

@nylen @tbuchok @FredKSchott

[nylen]

@dscape we just fixed this in #1253, can you do a quick test with master and specifying your multipart request like this:

{
    multipart : {
        chunked : false,
        data    : [ ... ]
    }
}

If that works for you, then I'll do a new release.

[nylen]

@dscape version 2.48.0 is out, that should fix your issue but let us know please.