v3 Roadmap

[xxczaki]

Note: v3 beta is now available on npm under the next tag

---

v3 Roadmap

General stuff

• BREAKING: Drop support for Node.js 4 and Node.js 6 (Update dependencies for node-fetch v3.x #489, Drop support for Node.js 4.x  #624)
• Update devDependencies (related to Update dependencies for node-fetch v3.x #489)
• When users clone(), automatically create streams with custom highWaterMark (When users clone(), automatically create streams with custom highWaterMark #386, PR open: Allow setting custom highWaterMark via node-fetch options (#386) #671)
• Bundle TypeScript types (Support for TypeScript without need of installing @types/node-fetch package) (feat: Migrated types #669)
• Replace Rollup with Microbundle (Plans to drop babel in v3 #643)
• Introduce linting to the project (Introduce linting to the project #303)
• Simplify Travis CI build matrix (related to Too big of a Travis CI matrix #431) - Dropping support for Node.js 4 & 6 will help
• Response.statusText should not default (Response.statusText should not default #578)
• Use w3c defined message for AbortError (Use w3c defined message for AbortError #657)
• URL is not encoded as UTF-8 (URL is not encoded as UTF-8 #245)
• Deprecate timeout in favor of AbortController (Should we deprecate timeout in favor of AbortController? #523)
• Data URI support (feat: Data URI support #659)
• Drop the optional dependency on encoding and move charset detection into a separate package. (In progress: Richienb/fetch-charset-detection (PR Open: refactor: Replace encoding with fetch-charset-detection. #694))
• Drop existing blob implementation code and use fetch-blob as dependency instead (v3 Roadmap #668 (comment))
• Fix stream piping (missing response stream error events #309, PR open: Fix stream piping #670)
• Fix stream resolving (Response promise resolves before response closes #642, Response.json() never resolves #665)
• Do not use constructor.name to check object (Do not use constructor.name to check object #667, PR open: fix: Better object checks #673)
• Properly forward request errors while piping response stream
• Convert Content-Encoding to lowercase (When server's Content-Encoding: GZip, response won't decompress #661, PR open: fix: Convert Content-Encoding to lowercase #672)
• More tests for data URI (v3 Roadmap #668 (comment))
• Drop credentials before redirection (When redirecting to a different host, strip Authorization header #274)
• Modernise the code behind FetchError and AbortError (v3 Roadmap #668 (comment))
• Propagate size and timeout to cloned response (Propagate size and timeout to cloned response #664)
• Replace the calls to url.parse and url.resolve with calls to new URL to use the new UTF-8 awareness and remove the utf8 package.
  • url.parse (refactor: Replace url.parse with new URL() #701)
  • url.resolve (e3cd4d2)
• Document types of fetch errors more thoroughly (Document types of fetch errors more thoroughly  #549)
• Should update host if there's a redirection (Redirect-following error (HTTP -> HTTPS + naked domain -> www subdomain) #570, PR Open: fix: Update host if there's a redirection #596)
• High priority doesn't correctly handle multiple content-type values (doesn't correctly handle multiple content-type values #783)

Additional:

• HTTP/2 support (Support HTTP/2 #342, PR open: chore: http2 support (WIP) #796)
• Plugin/middleware system (v3 Roadmap #668 (comment), PR open: feat: Add basic middleware functionality #677)
• WebStream API support (WebStream API support #387) - needs more discussion and standard.
• Support trailers (Support trailers #330) - trailer support has been removed (Consider removing trailers API whatwg/fetch#772).
• Caching (Offer caching #68) - would likely require opinionated implementation.
• Cookies (cookie jar #94) - would likely require opinionated implementation.

Other

• Enable Dependabot to automatically update dependencies if they pass tests and are within an appropriate version range.
• A way to back/sponsor the project (A way to back/sponsor the project #680)
• Create a final v2 release before releasing v3
• Update upgrade guide
• Release v1.x version of Richienb/fetch-charset-detection before release.
• Add this package to DefinitelyTyped/DefinitelyTyped/master/notNeededPackages.json since we now ship the typings.

---

cc @bitinn @jimmywarting @TimothyGu @jkantr @gr2m @Richienb

[bitinn]

Going through some items:

BREAKING: Drop support for Node.js 4 and Node.js 6 (#489, #624)
Update devDependencies (related to #489)

definitely

When users clone(), automatically create streams with custom highWaterMark (#386, PR open: #563)

probably don't want an API, clone() should probably just create both stream with larger highWaterMark so that isomorphic users don't run into issues that easily

Bundle TypeScript types (Support for TypeScript without need of installing @types/node-fetch package)

as long as there are people to review and maintain this one.

Rewrite to CommonJS & get rid of Rollup & Babel (#643)

I would love to, but need to think about potential impact?

Introduce linting to the project (#303)

I am fine with or without it, make rules more relax at first would be good.

Simplify Travis CI build matrix (related to #431) - Dropping support for Node.js 4 & 6 will help

definitely

Body conversion (#210)

jimmy can probably help you on that

Response.statusText should not default (#578)

yes, follow the spec on this

Use w3c defined message for AbortError (#657)

we should probably change to match browsers, but need to review how many error types we will end up with (quite a few, I reckon, in #549); error handling in browsers have always been a headache, and AFAIK Fetch just couldn't address it due to browser legacy and conventions.

URL is not encoded as UTF-8 (#245)

With older node dropped this should be easy to do

Deprecate timeout in favor of AbortController (#523)

I am at least a bit skeptical on this: AbortController sure gives a lot of control but timeout's ease of use seems hard to beat. I would think providing a helper module (outside of node-fetch, using AbortController) that emulate timeout is needed before we just drop timeout.

[Richienb]

@bitinn I can be a backer code reviewer. There are heaps of things in the roadmap that needs doing and I'm up for creating and reviewing PRs to resolve them. TLDR: Please make me a collaborator. 👍

[bitinn]

@Richienb sure, added! make sure you ask other for review before merging risky changes.

Also we should decide on whether we merge changes on 3.x branch or master branch; I was meaning to use 3.x but if we are already working on v3 release then master is fine too.

[Richienb]

@bitinn At the moment, I'm creating PRs against 3.x

[bitinn]

@Richienb let's go with 3.x then, unless @xxczaki or @jimmywarting has any objections

[Richienb]

@bitinn Sure. I've also just created the Discord server which I'll finish setting up within the next hour: https://discord.gg/hh8ZSw7

[bitinn]

Caching
Cookies

I wasn't planning to tackle these within node-fetch, mostly because I would like to make these features possible by having a simple plugin system, it was discussed in an old issue.

HTTP/2 support

I was waiting for node to provide some abstraction layer between HTTP/1 and HTTP/2 so that we can build upon it. Feels like a lot of work and a headache to maintain.

WebStream API

I am quite interested in this, but the ecosystem doesn't seem to be ready, can we easily convert between Node Stream and WebStream nowadays? Can we hide such conversion so that users' code is truly isomorphic? Lots of things to consider.

I was planning to allow users to provide a custom "Stream" implementation, much like a custom Blob or FormData, but that feels too much...

[Richienb]

@bitinn How are you envisioning the simple plugin system to work?

For WebStreams, wouldn't https://www.npmjs.com/package/node-web-streams cut the cake?

[Richienb]

@bitinn In the Discord server, do we need a feed holding all of the activity in this repository? If so, add a webhook for all events to https://discordapp.com/api/webhooks/619919240027308042/YpxDdMxaQ3oOgtoZfkAWovJ3A4pztZFWBD1pF0mU6rlQpz0vaRa-J1_sPMIHO24c0YtX/github in the repo settings.

[xxczaki]

@bitinn

probably don't want an API, clone() should probably just create both stream with larger highWaterMark so that isomorphic users don't run into issues that easily

I agree. It's also worth mentioning, that there is a 80$ bounty on this issue

as long as there are people to review and maintain this one.

I'm happy to maintain types, so that shouldn't be a problem 😄\

I am fine with or without it, make rules more relax at first would be good.

Sure, we can use ESLint wrapper, like xo to get some reasonable defaults and turn off rules we don't like.

---

Let's leave HTTP2 support & WebStream API and wait for them to be fully implemented in Node.js Implementing HTTP/2 would indeed be a hard task (see for example sindresorhus/got#167).

[bitinn]

@Richienb I haven't thought about it too much to be honest, maybe we don't need plugins at all, just allow users to set a fetch.CookieJar might be enough. AFAIK only caching and cookie require such external dependencies.

[Richienb]

@xxczaki

I'm happy to maintain types, so that shouldn't be a problem 😄\

In that case, we can go ahead and merge #669

[xxczaki]

@Richienb Done 😄

[Richienb]

@bitinn @xxczaki I'm currently working on adding the xo linter to the project. What type of spacing are we going for? Spaces? Tabs?

For now, I'll go with 4 spaces.

[xxczaki]

@Richienb Tabs are my favorite

[xxczaki]

@Richienb I will start working on:

• Rewrite to CommonJS & get rid of Rollup & Babel
• Drop support for Node.js 4 and Node.js 6
• Update devDependencies

As soon as you add xo 😄

[Richienb]

@xxczaki It turns out, not using a linter for 4 years may not have been a good choice. XO is spitting out 50 or so errors that can't be automatically fixed and I'm slowly grinding through them, adding ignore rules and fixing the fixable.

[gr2m]

I would suggest to add an automated release process to the list, based on 3 commit message conventions (fix: ... & feat: ... in commit subject, BREAKING CHANGE: ... in body). I would not be able to keep as many projects up-do-date as I do without the combination of @greenkeeperio & @semantic-release.

I can help with both Greenkeeper & semantic-release setup and if there are any problems that would arise from it in future. I worked on Greenkeeper in the past and I am a maintainer at semantic-release myself.

[gr2m]

Regarding

Introduce linting to the project

I would strongly suggest to use a zero-config linter, such as https://standardjs.com/ or https://prettier.io/ (without custom config). Nobody will ever be happy with any linting setup, but at least we can avoid having discussions about what linting rule to add or change, and focus on the code instead.

[xxczaki]

@xxczaki It turns out, not using a linter for 4 years may not have been a good choice. XO is spitting out 50 or so errors that can't be automatically fixed and I'm slowly grinding through them, adding ignore rules and fixing the fixable.

I will be happy to help with some of the errors, if you want to 😄

[xxczaki]

I would suggest to add an automated release process to the list, based on 3 commit message conventions (fix: ... & feat: ... in commit subject, BREAKING CHANGE: ... in body). I would not be able to keep as many projects up-do-date as I do without the combination of @greenkeeperio & @semantic-release.

I can help with both Greenkeeper & semantic-release setup and if there are any problems that would arise from it in future. I worked on Greenkeeper in the past and I am a maintainer at semantic-release myself.

I will need to think about that, semantic-release seems really nice (I've actually used it in one of my projects in the past), but I'm not sure about Greenkeeper... Have you ever used Dependabot? It has some additional features and was also recently aquired by Github, so it might be a better option.

[gr2m]

I’ve always used greenkeeper due to my affiliation to it, but in the end they all do the same, so I really don’t mind which we end up using, as long as we have one. As we already have 100% test coverage we will be able to merge dependency updates quickly and with confidence, no matter the service that sends the PRs.

[gr2m]

I've found that timeout was still documented in the README, I started a PR at #877.

While investigating, I found other options that are defined in the RequestInit but are not documented as node extensions:

• counter
• hostname
• port
• protocol

can we add them to the node-only extensions?

[ryanflorence]

Any plans for let data = await request.formData() ? Been cruising through issues and can't find anything.

[tinovyatkin]

Now, when the current LTS is 14.x, I believe it's reasonable to drop < 12.18 support in the v3 release? We can drop quite a few legacy logic then...

[tekwiz]

@tinovyatkin 12.x is still under maintenance support for another 16 months (and 10.x is also for another 4 months). I'm afraid that if we drop support for <12.18, we will just end up with a bunch of irritated users with lots of "wontfix" issues. If the intent is just to remove the logic branches, I think the public relations benefit of leaving them is greater than the cost.

[mariusa]

node 10.x is no longer maintained; 12.x users can upgrade to >=12.18 (they do this anyway to get security fixes), they aren't forced to jump to 14.x

[gr2m]

I agree with @mariusa. v2 has been around for a long time and is stable. We can continue to accept bug fixes for v2 for a while.

With v3, we should drop support for Node < 12.8 and build node-fetch as native ES Module, without any transpilation to CJS, just make a clear break, similar to what Sindre laid out here: https://blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77

[xxczaki]

I'm back and happy to make a PR to use pure ESM 😄

[gr2m]

Looks like @Richienb and @jimmywarting are in favor, too

Any thoughts @bitinn?

[bitinn]

I am onboard, @xxczaki you can do it.

My hope is v3 stable gets out this year :)

Also note there are some outstanding PRs, fixing known issues is probably more useful than a rewrite.

(but I understand the desire to have a clean break.)

[xxczaki]

@node-fetch/core @node-fetch/reviewers and others,

I opened a pull request which includes the ESM transition and more. Feedback is appreciated, as merging this will allow for the release of 3.0.0-beta.10 which will include some important bug fixes.

[jimmywarting]

the v3 beta have been around for quite a while now, can't make a release happen soon?

[gr2m]

I'm sorry I didn't have the time to review @xxczaki pull request for the ESM transition yet. I do have it on top of my work todo list though, I'll very likely get to it this coming week.

[jimmywarting]

Good news!

WHATWG streams have landed in NodeJS (as experimental)
nodejs/node#39062

Think we should start using it whenever possible, might be worth looking into adding whatwg stream polyfill now!
It would be a major breaking change. Maybe start planing for v4 straight away

---

Wow, githubs new markdown to display the full title of other issues/pr really made a mess

[jimmywarting]

ESM has landed! 🚀

This issue have become quite long to scan through.
Is it worth shipping 3.0.0-beta.10 now, or should we fix any other issues?

Going from beta.9 to current version is a major breaking change from going from cjs to esm that i feel that we can safely release v3 without having to make any additional breaking changes later (except when we start using real whatwg streams - but will probably hold of from this a tiny bit longer).
Feature PR will mostly only need minor/patch fixes.

One thing that worries me now is that ppl that are already using 3.0.0-beta.9 with require() would start breaking if we released 3.0.0-beta.10? maybe should just go for 3.0.0?

[gr2m]

One thing that worries me now is that ppl that are already using 3.0.0-beta.9 with require() would start breaking if we released 3.0.0-beta.10?

breaking changes are ok for pre-releases such as -beta.X, I wouldn't worry about it. People who have 3.0.0-beta.9 installed will not automatically get 3.0.0-beta.10 when they run npm install, it needs an explicit change in the package.json.

[jimmywarting]

People who have 3.0.0-beta.9 installed will not automatically get 3.0.0-beta.10 when they run npm install

Good to know. Should we publish the next beta version now then?

After that I say we hold off for a while with any new features/PR and only fix bugs that might arise.
If there is no major issue I think that it has gotten time that we start releasing v3 - it have been put off for such a long time now and there are many new feature/bug fixes since v2.

(unless someone else really need/want something else to be fixed before we ship v3)

[jimmywarting]

beta.10 release!

Entering idle/maintenance mode for 2-4 week 🙂
let's see how it plays out. and maybe we can release it as stable afterwards

[styfle]

It looks like the size increased quite a bit

• node-fetch@3.0.0-beta.9
• node-fetch@3.0.0-beta.10

Seems like most of that is from web-streams-polyfill.
Is that needed for node-fetch?

[jimmywarting]

@styfle

b/c NodeJS added native support for whatwg:streams we thought: Ok, so whatwg streams are actually going to become useful in NodeJS landscape.
So i was thinking: Ok so if we are going to be a references a nodejs implementation then we have to do everything nodejs is doing so there isn't any differences between implementations

so: fetch-blob was the first to adapt web-streams-polyfill
now i'm also thinking that Body.body should also be a whatwg:streams in some feature release, so yea, we are going to depend on it from now on.

NodeJS is also going to ship fetch in core one day. and we wish to be the polyfill that references nodejs implementation so it can be backward compatible until everyone starts using nodes (upcoming) built in fetch impl that means we should also be using whatwg stream for compatibility reasons.

#1217 tracks the size issue, and @MattiasBuelens is working on reducing the size here: MattiasBuelens/web-streams-polyfill#83

Is that needed for node-fetch?

short answer: yes
in Blob.prototype.stream and response.body