Reponse.text() hangs if you've cloned the response

[smikula]

Reproduction

Steps to reproduce the behavior:

const fetch = require('node-fetch');

fetch('https://github.com').then((response) => {
    const clone = response.clone();

    setTimeout(() => {
        clone.text().then((text) => {
            console.log('Got clone text.');
        });
    }, 10000);

    response.arrayBuffer().then((text) => console.log(text));
});

Expected behavior

I expect the text of the response to be printed to the console immediately. Instead, the response.text() promise only resolves after the timeout (when clone.text() is called). If you create the clone but never consume it's body, the response text never gets printed to the console.

Your Environment

software	version
node-fetch	2.6.1
node	14
npm
Operating System	Windows 10

Additional context

[benmccann]

See https://github.com/node-fetch/node-fetch#custom-highwatermark

[halofe]

node v16.14.0
"node-fetch": "^3.2.10"

import fetch from "node-fetch"

async function req(url) {
  const resp = await fetch(url, {
    highWaterMark: 101024 * 1024
  }).then(x => x.clone())
  console.log(url, await resp.text())
}
await req('https://jsonplaceholder.typicode.com/todos/1')
await req('https://baidu.com')

highWaterMark option seems not work, only the first request get returned

[iscekic]

Running into a similar-sounding issue using ky-universal: sindresorhus/ky-universal#46

After response.clone(), .json() and others never resolve.

[john-doherty]

I can confirm, this is still an issue - regardless of setting the highWaterMark

[john-doherty]

Super frustrating situation:

• My project uses node v14.16.0 with lots of require statements
• Because of that, I'm limited to version 2.x of node-fetch
• Which means I can't use the highwatermark feature in version 3.x

I've tried many ways to mix es6 import with commonjs but couldn't get it to work, so I'm stuck at 2.x.

Am I right in thinking the only solution from here is to upgrade my entire app to use imports? 😢

[jimmywarting]

@john-doherty have you tried using async import? this works in cjs without having to switch out everything to using import

const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args))

fetch('https://httpbin.org/get')
  .then(res => res.text())
  .then(console.log)

I just tried this with:

npm init -y
npm i node-fetch@3
echo "
const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args))

fetch('https://httpbin.org/get')
  .then(res => res.text())
  .then(console.log)
" > index.js
node index.js

and it works just fine.
This approach makes loading node-fetch by using lazy loading tech. so it's only loaded when needed. so boot up time might be faster.

Is there any particular reason why you could not just update to newer NodeJS version (18+) that have fetch builtin and maybe just use that instead?

[jimmywarting]

the fact that you stumble upon this issue is a sign that you are doing things wrong.
a cloned (tee'ed) response should both be consumed at the same time in best senario.

eg: one response should go to the cache and the other goes towards parsing/using the response.
if you are waiting for just one of the responses to consume the entire body before you even start consuming the cloned response is a bad practices.

res = await fetch(url)
clone = res.clone()

// Bad
const text = await res.text()
const ab = await clone.arrayBuffer()

// ok:
const [text, ab] = await Promise.all([ 
  // start consuming both at the same time
  res.text(), 
  clone.arrayBuffer() 
])

...if it's a case of parsing json and if it fails then and then should fallback to reading it as plain text, then you should consider to not depend on .json(). instead use .text() and then try parsing it with JSON.parse (possible with a try/catch)

[john-doherty]

@john-doherty have you tried using async import? this works in cjs without having to switch out everything to using import

const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args))

fetch('https://httpbin.org/get')
  .then(res => res.text())
  .then(console.log)

I just tried this with:

npm init -y
npm i node-fetch@3
echo "
const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args))

fetch('https://httpbin.org/get')
  .then(res => res.text())
  .then(console.log)
" > index.js
node index.js

and it works just fine. This approach makes loading node-fetch by using lazy loading tech. so it's only loaded when needed. so boot up time might be faster.

Is there any particular reason why you could not just update to newer NodeJS version (18+) that have fetch builtin and maybe just use that instead?

Yes, I tried that, but it did not work... it's not possible to upgrade the project, too many dependencies that would need to be upgraded/tested etc

[john-doherty]

the fact that you stumble upon this issue is a sign that you are doing things wrong. a cloned (tee'ed) response should both be consumed at the same time in best senario.

eg: one response should go to the cache and the other goes towards parsing/using the response. if you are waiting for just one of the responses to consume the entire body before you even start consuming the cloned response is a bad practices.

res = await fetch(url)
clone = res.clone()

// Bad
const text = await res.text()
const ab = await clone.arrayBuffer()

// ok:
const [text, ab] = await Promise.all([ 
  // start consuming both at the same time
  res.text(), 
  clone.arrayBuffer() 
])

...if it's a case of parsing json and if it fails then and then should fallback to reading it as plain text, then you should consider to not depend on .json(). instead use .text() and then try parsing it with JSON.parse (possible with a try/catch)

I'm building a disk caching layer over the top of fetch; it needs to clone the response so it can be saved, and then return the response to the caller untouched, so it's not possible to execute both in a Promise.all

[john-doherty]

To provide some context, I'm trying to create a server-side version of offline-fetch; which I've had running in production without issue for some time:

// this code is from client side offline caching offline-fetch
res.clone().text().then(function (content) {

    var contentToStore = JSON.stringify({
        url: url,
        status: res.status,         // store the response status
        statusText: res.statusText, // the response status text
        contentType: contentType,   // the response content type
        content: content,           // the body of the response as a string
        storedAt: Date.now()        // store the date-time in milliseconds that the item was cached
    });

    // store the content in cache as a JSON object
    storage.setItem(cacheKey, contentToStore);
});

// original response returned untouched
return res;

[john-doherty]

I worked around it by reading the string:

var text = await res.text();

And then reconstructing the response object:

res = new Response(text, {
    status: res.status,
    statusText: res.statusText,
    headers: Object.fromEntries(res.headers.entries()),
    url: url
});

Setting highWaterMark had no impact for me. Hope this helps

[handlebauer]

I've run into this issue as well, although using node's native fetch. Interestingly, it was only json() in conjunction clone() I was running into problems with (text() worked fine 🤷‍♂️) .

My way around this was proxying the incoming response something like:

/**
 * @param {Response} response
 */
export const cloneResponse = response => {
  /**
   * @type {{ text: Promise<string>, json: Promise<any> }}
   */
  const cache = {
    text: undefined,
    json: undefined,
  }
  return new Proxy(response, {
    get: (response, prop, receiver) => {
      if (prop === 'url') return response.url

      if (prop === 'text') {
        if (cache.text) return () => cache.text
        cache.text = response.text()
        return () => cache.text
      }

      if (prop === 'json') {
        if (cache.json) return () => cache.json
        cache.json = response.json()
        return () => cache.json
      }

      return Reflect.get(response, prop, receiver)
    },
  })
}

[silverwind]

Another variation of this hang occurs when trying to consume the clone body first and the body size is greater than highWaterMark. It will completely hang the v8 thread after attempting to consume the body, so the code after the first .text() is never reached.

// import fetch from "node-fetch";
import {fetch} from "undici";

setTimeout(() => console.log(Math.round(performance.now()), "exit"), 5000);

const res = await fetch("https://github.com");
console.log(Math.round(performance.now()), "fetch done");
await res.clone().text();
console.log(Math.round(performance.now()), "clone done");
await res.text();
console.log(Math.round(performance.now()), "res done");

with node-fetch:

202 fetch done
5069 exit

with undici is works as expected:

231 fetch done
290 clone done
291 res done
5098 exit