v4 Roadmap

[jimmywarting]

General stuff

v3 may have been short lived, cuz of how long time we dragged out on esm.
We already have a milestone for v4 with few issues in it.
NodeJS v12 will soon end its LTS version in April, so i was thinking of dropping support any prior version and remove all the hacks we have around it in the current branch.

I was thinking about setting the lowest supported node version to v14.17.0
not 14.13.1 as we have today that is meant for ESM
why? cuz http.request supports signal in v14.17.0
then we can also use finalizers that got introduced in v14.6.0 and cleanup the response (and the cloned res) when it's forgotten and not consumed

Is there any bug/issue/features we would like to finish or complete before we branch out of v3 and start working on v4?

• General
  • Create a v4 update guide
  • create some "next/beta" release on npm
• signal
  • Start depending on NodeJS own built in AbortController
  • Every request should have a new associated AbortSignal #1439
  • Don't check if AbortSignals tag name match EventTarget #1469
  • forward this signal to http.request(url { signal })
  • remove the abort & finalize logic and use this 👆 abort signal to abort the request.
• Headers
  • chore: replace headers class #1484
  • remove some own test around headers (as ☝️ runs against wpt, + some own misc test)
• whatwg streams
  • start depend on web-stream-polyfill (as fetch-blob have start depending on it)
  • Impl whatwg streams #1213
  • Allow request take AsyncIterable body? whatwg/fetch#1291
  • convert node's incoming http stream and the unpacking br, deflate logic to a whatwg stream
  • set .body to be a whatwg stream
• URLSearchParam
  • remove isURLSearchParam and only accept global URLSearchParam introduced in v10
• package
  • Drop support for v12 and set target version to ^ 14.17.0 || >=16.0.0"?
  • fix(package): Add exports field in package.json #1347
• cleanup
  • Remove async iterator fix when Node 12 goes EOL #1223
  • remove the current stream clone logic and use ReadableStream.prototype.tee instead
  • chore: completely remove support for form-data package #1486
  • chore: completely remove .buffer() #1485
  • remove support for size option (fixes Deprecate size option #1438)
  • make use of finalizers to cleanup
  • remove some old test cases
  • We can also start using optional chaining

cc @bitinn @TimothyGu @jkantr @gr2m @Richienb @ronag @LinusU

[ronag]

If we break out undici headers and form data into separate packages. Would you consider using that?

[jimmywarting]

hmm, yea, but they are already broken out into separate packages

• Blob & File (which also includes fileFrom(path) to be backed up by the filesystem
• FormData
• Headers

i would rather want to see it landing into NodeJS core instead

[LinusU]

Looks like a great list!

• Re. fix(package): Add exports field in package.json #1347, I think that we should remove the main field as well.
• Node.js v14.18.0 added Blob directly to core, can we set the target to ^14.18.0 and use that?
  • A quick glance says we cannot, but maybe we could make a new major version of fetch-blob that extends the builtin, and just overrides/adds the small differences...
• I would love to see us sharing more code with undici!

[jimmywarting]

Blob.stream was added in: v16.7.0

[chris-kruining]

Anything I can do to help v4 become a reality? I'm working on a grpc lib and the sooner the whatwg streams are a reality the better that is for me 😊

[bitinn]

Yes, we are open to whatwg stream in v4, Jimmy will probably lead that effort. We are open to experimental PR on it. I am not certain fetch is the best API for rpc calls, best make the spec covers all you requirement, as we do intend to follow the spec more closely.

…

Sent from my iPhone

On Jan 21, 2022, at 15:48, Chris Kruining ***@***.***> wrote:  Anything I can do to help v4 become a reality? I'm working on a grpc lib and the sooner the whatwg streams are a reality the better that is for me 😊 — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[jimmywarting]

Anything I can do to help v4 become a reality? I'm working on a grpc lib and the sooner the whatwg streams are a reality the better that is for me 😊

Umh... i would like to finish some issues that are possible to solve before we jump onto the v4 wagon. it's still some few mounts left to v12 goes EOL. and i'm not sure whether or not i want to abandon the v3 just yet.

But there could be things we could work on towards making whatwg streams possible to use in some ways.
for starters: we could make it possible for the Request/Response constructor acceptable to any async iterator that yields Uint8Array.

import { Response, Blob } from 'node-fetch'

// i think this works:
const blob = new Blob(['abc'])
await new Response(blob).text() === 'abc'

// but this one dose not
const stream = blob.stream() // returns a web readable stream
await new Response(stream).text() === 'abc'

We need to improve the writeToStream to be able to write to node's http.request using whatwg streams, node streams and or using async iterators.

Some suggested change could be:

/**
 * Write a Body to a Node.js WritableStream (e.g. http.Request) object.
 *
 * @param {Stream.Writable} dest The stream to write to.
 * @param obj.body Body object from the Body instance.
 * @returns {Promise<void>}
 */
export const writeToStream = async (dest, {body}) => {
  if (body === null) return dest.end()
  
  // Body is async iterable (node stream, whatwg stream or any iterable that yields uint8array
  for await (const chunk of body) {
    dest.write(chunk)
  }
}
dest.end()

I have tried making a change like this before - but it broke some tests.

[chris-kruining]

@bitinn

I am not certain fetch is the best API for rpc calls, best make the spec covers all you requirement, as we do intend to follow the spec more closely.

That's actually even better, I am making this lib for the browser, so if node has a spec compliant fetch implementation that is a win for me.

@jimmywarting

We need to improve the writeToStream to be able to write to node's http.request using whatwg streams, node streams and or using async iterators.

That seems doable, node stream, whatwg stream are both also async iterable. So updating the constructor of Body should do the trick right?. Shall I make a PR that updates the body class to support (async) iterables?

[jimmywarting]

We can't exactly change the body class to be a whatwg stream or a async iterator in v3
in v3 the body should still be a node:stream

I think the easiest corse of action is to just look in body.js file if the body is of type whatwg:stream then convert it to a node stream using something like

if ( isNodeStream ) body = body
if ( isIterable ) body = stream.Readable.from(body)

But we can't just go a head and convert anything that is iterable either...
someone might do: await new Response((['jon', 'bob', 'alice'])).text() for some reason, the FormData is iterable also.
as long as whatwg/fetch#1291 is not speced and things can fallback to being casted to strings, then i think i would like to do:

if ( isNodeStream ) body = body
if ( isRedableStreamLike ) body = stream.Readable.from(body)

there is also stream.Readable.fromWeb that we can use. but that got added later in v17.0.0 ( I also don't know if that one works with web streams polyfill )

[jimmywarting]

I would like to have a isomorphic-streams package that it's sole purpose is to retrieve some kind of streams that we can use. and also have it in body node-fetch and inside of fetch-blob package (maybe it should also try to suppress the experimental warning message when importing the node:streams/web

I found this one earlier:
https://www.npmjs.com/package/isomorphic-streams

but it only supports NodeJS v16

We need something that can fall back to web stream polyfill...
I know that Mattias have some plans already to extend the built in stream whenever it's available so that streams can be transferable via workers and also work together with a native streams MattiasBuelens/web-streams-polyfill#20

[jimmywarting]

just realized something...
AbortSignal may have arrived in 14.17 but you have launch Node.js using the --experimental-abortcontroller flag. 😕
on v15 it was globally available and taken out of experimental

I wonder what that means for http.request(url, {signal}) - if it would be able to function with a polyfil.

maybe we should bring in a polyfill for time being...
could use a conditional import

const AbortController = globalThis.AbortController || await import('polyfill')

[chris-kruining]

I realise this might not be useful at all, but in the off-chance it is:

In order to have http2 and async iterables, I implemented a VERY minimal 'version' of this lib myself. feel free to take a peek and steal any code you want from this for the 3.x and 4.x milestones

(motivation for a custom implementation: this lib doesn't currently support http2 and fetch-h2 isn't compatible with the types in TS)

https://github.com/FYN-Software/grpc/blob/main/src/fetch.ts

[bitinn]

Thx, one thing that would be very useful for the nodejs community: an abstraction on top of http1 and http2, node-fetch can call into that API instead of http module.

If anyone is willing to take the task, I would love to open a repo at node-fetch org for him/her to lead, perhaps even throw in our open collective donation as bonus :)

[chris-kruining]

I just discovered that nodejs 18 is getting native fetch support. so, is it worth much to put a lot of effort into this lib still? I have no clue how far node 18 is off yet. but I feel like it's is a massive waist of good effort from the contributors if the work is obsolete in -I am guessing- a couple of months

[jimmywarting]

I would also like to start using built in node's fetch in v18
but many developers still supports LTS version (12 atm) and it takes years for everyone to want to update.
i wish it had the same release cycle as chrome every 6week bind the scenes with auto updates.

[jimmywarting]

@agrohs what's your reason for 👎 ?

[dnalborczyk]

I have no clue how far node 18 is off yet.

@chris-kruining https://nodejs.org/en/about/releases/

[Richienb]

If all goes to plan, node-fetch will no longer be needed by 30/04/2024, the date when Node.js v16 is EOL. Until then, we'd want to backport whatever v18 does to implement fetch.

[regseb]

Can the pull request #1473 be included in v4? It changes the default value of the Connection header by replacing close by keep-alive.

[thetayloredman]

you might not want to release a new version. Node has built-in fetch @ node:fetch starting in v18.

[jimmywarting]

you might not want to release a new version. Node has built-in fetch @ node:fetch starting in v18.

There are things even the NodeJS impl is missing... like parsing application/multipart for instances...
Also, NodeJS blob impl isn't actually that good. there are many things i could nag about that makes me want to use fetch-blob rather than NodeJS own impl.

Also not everybody can update to NodeJS v18 yet

[ravshansbox]

I'm quite surprised that node team decided to go with a different implementation

[loren138]

Is there an official statement on native fetch in node 18 vs this library?
Or any statements on of the continued maintenance of this library?

Stability 1 (experimental) from the native implementation https://nodejs.org/dist/latest-v18.x/docs/api/globals.html#fetch has my team worried about moving to it, but at the same time, we don't want to move to this if you may be phasing out support in September 2023 when node 16 reaches EOL.

[jimmywarting]

I would not worry about it to much. their fetch impl is really grate and even more spec compliant then our own. it solves some thing more correctly then what we are doing in node-fetch

the only culprit could maybe be that it dose not support using a own custom http agent or that some compressed responses aren't able to be unpacked ^{[1 2]}. one has been resolved in the undici package already by me and have been published to npm already, i guess it takes a tiny bit longer for things to those changes to land in NodeJS core cuz of their release cycle.

The other thing could maybe be that you lose the functionality of getting everything that is inherited from fetch-blob namely createBlobFrom(path, 'text/plain') and using that along with FormData among other friends. but for that you could import fetch-blob directly yourself cuz undici support cuz they do doing duck-type checking rather than strict instanceof checks. and if you are using NodeJS v20 then there is even a built in fs.openAsBlob(path, { type })

Other then that our long term plan is for node-fetch to eventually take it corse and die out when it's no longer needed fetch is provided by NodeJS core natively.

undici have also imported all our our own test to make sure that undici haven't forgotten anything. few of them are disabled (if you search for xit under https://github.com/nodejs/undici/tree/main/test/node-fetch (but that was only b/c we did some things incorrectly) like having node:streams instead of web streams on response.body among other things.

---

one thing i might have planned on doing for v4 is just simply re-export everything from undici (to have node v16 support) where as when v16 reaches EOL we would then just only re-export the builtin instead. at which point it's just a completely worthless thing to have as a dependency. may not even do that either. maybe will just add a deprecation notice when v16 reach EOL in sept.

• node-fetch v4.x: only support v16.7+ an re-export undici fetch
• node-fetch v5.x: only support v18+ an re-export global fetch (or deprecate this package)

it might depend on if we are able to find other useful usecases for node-fetch in the feature or what we will do next after this if anything at all... maybe we wanna build a variant that can support cookie jar. progress event, patching response.blob() / response.formData() so that files are stored on a temporary location on the disk instead of holding it in memory, or if we want to build a CacheStorage that is able to cache responses and can handle response caches and so forth... the feature of node-fetch is unclear atm.

[isnolan]

node-fetch is really great. Although the fetch api has been added since node.js v18, I chose node-fetch because of the need for proxy. This is the necessity of node-fetch.

Also, are there any plans to add support for fingerprint, which can be setup with tls/ja3?

bless