src: do not reuse HTTPParser #25094

Drieger · 2018-12-17T18:55:53Z

Change resource being used, previously HTTParser was being reused. We are now using IncomingMessage and ClientRequest objects. The goal here is to make the async resource unique for each async operation

Refs: #24330
Refs: nodejs/diagnostics#248
Refs: #21313

Co-authored-by: Matheus Marchini matheus@sthima.com

I continued @mmarchini work on #24330 and finished what was missing. I also added a test based on @AndreasMadsen comment nodejs/diagnostics#248 (comment)

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

mmarchini · 2018-12-17T19:14:33Z

CI: https://ci.nodejs.org/job/node-test-pull-request/19624/ ❌

mmarchini · 2018-12-17T19:14:50Z

cc @nodejs/async_hooks @nodejs/diagnostics

mcollina

LGTM

Drieger · 2018-12-17T19:55:06Z

The problem with linter is that common is being imported but not used, but when I do not import common linter error is that it is mandatory. Any ideas how to solve this one?

richardlau · 2018-12-17T19:57:12Z

The problem with linter is that common is being imported but not used, but when I do not import common linter error is that it is mandatory. Any ideas how to solve this one?

Just write:

require('../common');

or alternatively use common (e.g. common.mustCall).

Flarna · 2018-12-17T20:00:26Z

If I understand this correct the resource type is still HTTPPARSER but depending on incoming/outgoing the resource passed in init differs. I think this is confusing.

Wouldn't it better to have two resource types instead?

mscdex · 2018-12-18T00:23:06Z

Perhaps the commit message could be more clear. 'do not reuse HTTPParser' makes it sound as if parser objects are not being reused, when in fact it's the associated async resource that's no longer being reused (if I understand correctly).

test/async-hooks/test-httparser-reuse.js

AndreasMadsen

Thanks for spending time on this.

src/async_wrap.cc

Drieger · 2018-12-19T12:13:19Z

@Flarna maybe we could use IncomingMessage and ClientRequest instead? What do you think?

Flarna · 2018-12-19T12:57:35Z

Not sure here; maybe prefix with Http as these names are quite generic in the global AsyncHooks scope.

Flarna · 2018-12-19T13:03:45Z

lib/_http_server.js

-  parser.reinitialize(HTTPParser.REQUEST, parser[is_reused_symbol]);
+  parser.reinitialize(HTTPParser.REQUEST,
+                      parser[is_reused_symbol],
+                      server[kIncomingMessage]);


Isn't this reusing the IncomingMessage constructor function instead of the concrete instance? If I understand the code correct the IncomingMessage instance is created in parserOnHeadersComplete.
If I'm correct I wonder why the dedicated test-httparser-reuse.js is not detecting this.

@Drieger This one is still open and if I'm correct this is a no go.

You are right @Flarna I'll investigate this more in depth.

@Flarna kIncomingMessage is really a function as you said. I fixed the test to catch this.
I will also create a new object to be the resource for the server. Some other changes will be necessary as well. I will update the PR as soon as I this is working.

Drieger · 2019-01-09T10:38:44Z

Not sure here; maybe prefix with Http as these names are quite generic in the global AsyncHooks scope.

@Flarna I just updated the PR, adding two new providers HTTPINCOMINGMESSAGE and HTTPCLIENTREQUEST, could you take a look if that is what you had in mind?

I'll check why is the CI failing.

Flarna · 2019-01-09T11:01:57Z

Yes, looks ok. I think an alternative would be HTTP_REQUEST/HTTP_RESPONSE like in code which would point to what happens instead naming the resource type used (currently).

I would love if others give also their comments on these names as they are actually public API (even it's just in experimental part).

Is HTTPPARSER still needed?

https://github.com/nodejs/node/blob/master/doc/api/async_hooks.md needs also some update once agreement on the names is reached.

Drieger · 2019-01-09T11:13:12Z

You are right, I think HTTPPARSER is not necessary anymore. I'll update doc as soon as the "final" name is chosen.

Drieger · 2019-01-10T12:31:08Z

Fix test issues!
I'll update commit message after all changes have been approved.

/cc @mmarchini @mcollina @AndreasMadsen @richardlau @mscdex could you check again? What do you think about the naming suggested by @Flarna?

Drieger · 2019-01-14T18:47:00Z

I changed the resource in the _http_server.js file and fixed the test that was not able to capture the previous situation (function was being reused). I think it would be a good idea to run benchmarks if everyone agrees with the current implementation.

Obs.: Documentation and commit message will be updated if everything is ok.

mcollina · 2019-01-15T04:30:38Z

lib/_http_server.js

+  // Force reinitilalization to make sure AsyncReset is called
+  parser.reinitialize(
+    HTTPParser.REQUEST,
+    true,


I think we should remove this parameter, or is it used somewhere else?

Removed parameter and renamed the method.

mcollina · 2019-01-17T12:38:06Z

CI: https://ci.nodejs.org/job/node-test-pull-request/20179/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1719/

mcollina · 2019-01-17T12:38:58Z

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/287/

mcollina · 2019-01-19T08:45:15Z

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/288/

(again, the previous one errored).

Drieger · 2019-03-06T20:25:54Z

I think I addressed all previous comments, any other comments on this? @nodejs/diagnostics @nodejs/async_hooks

mcollina

LGTM

mcollina · 2019-03-07T14:35:21Z

I've tagged this semver-major because of the impact it can have on async_hooks users. It's not technically required because async_hooks are experimental, but they are widely used.
I'm fine in being semver-minor with a dont-land-on-8.x and dont-land-on-10.x label as well.

mmarchini · 2019-04-22T18:51:16Z

Landing

mmarchini · 2019-04-22T18:53:45Z

Landed in ece5073 🎉 🎉 🎉

A huge thanks for everyone who reviewed and worked on this!

Change resource being used, previously HTTParser was being reused. We are now using IncomingMessage and ClientRequest objects. The goal here is to make the async resource unique for each async operatio Refs: #24330 Refs: nodejs/diagnostics#248 Refs: #21313 Co-authored-by: Matheus Marchini <mat@mmarchini.me> PR-URL: #25094 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benedikt Meurer <benedikt.meurer@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Flarna · 2019-04-22T21:00:46Z

I hoped that this one fixes #26961 but actually it didn't. I think that init hook is not called in case a Parser is reused since this change.

Fix some issues introduced/not fixed via #25094: * Init hook is not emitted for a reused HTTPParser * HTTPParser was still used as resource in init hook * type used in init hook was always HTTPINCOMINGMESSAGE even for client requests * some tests have not been adapted to new resource names With this change the async hooks init event is emitted during a call to Initialize() as the type and resource object is available at this time. As a result Initialize() must be called now which could be seen as breaking change even HTTPParser is not part of documented API. It was needed to put the ClientRequest instance into a wrapper object instead passing it directly as async resource otherwise test-domain-multi fails. I think this is because adding an EventEmitter to a Domain adds a property 'domain' and the presence of this changes the context propagation in domains. Besides that tests still refering to resource HTTPParser have been updated/improved. Fixes: #27467 Fixes: #26961 Refs: #25094 PR-URL: #27477 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

Avoid that destroy hook is invoked twice - once via `Parser::Free()` and once again via `Parser::Reinitialize()` by clearing the async_id in `EmitDestroy()`. Partial backport of nodejs#27477, a full backport would require also nodejs#25094 which has a dont-land-on-v10.x label on it. Fixes: nodejs#26961

Avoid that destroy hook is invoked twice - once via `Parser::Free()` and once again via `Parser::Reinitialize()` by clearing the async_id in `EmitDestroy()`. Partial backport of #27477, a full backport would require also #25094 which has a dont-land-on-v10.x label on it. Fixes: #26961 Backport-PR-URL: #27986 PR-URL: #27477 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. labels Dec 17, 2018

mmarchini mentioned this pull request Dec 17, 2018

WIP http_parser: don't reuse HTTPParser #24330

Closed

3 tasks

Drieger force-pushed the avoid_reuse_of_HTTPParser branch from 368187e to 5d9eebd Compare December 17, 2018 19:13

mcollina approved these changes Dec 17, 2018

View reviewed changes

Trott reviewed Dec 18, 2018

View reviewed changes

test/async-hooks/test-httparser-reuse.js Show resolved Hide resolved

AndreasMadsen reviewed Dec 18, 2018

View reviewed changes

src/async_wrap.cc Show resolved Hide resolved

Flarna reviewed Dec 19, 2018

View reviewed changes

mcollina reviewed Jan 15, 2019

View reviewed changes

mcollina mentioned this pull request Jan 28, 2019

process: improve nextTick performance #25461

Merged

2 tasks

mcollina added the semver-major PRs that contain breaking changes and should be released in the next major version. label Mar 7, 2019

mcollina approved these changes Mar 7, 2019

View reviewed changes

mmarchini closed this Apr 22, 2019

refack mentioned this pull request Apr 23, 2019

benchmark: fix http/bench-parser.js #27360

Closed

3 tasks

Flarna mentioned this pull request Apr 23, 2019

test: replace HTTPPARSER by new resource names #27372

Closed

3 tasks

This was referenced Apr 28, 2019

Asynchook is using destroyed AsyncId for new executions #27467

Closed

async_hooks: fixup do not reuse HTTPParser #27477

Closed

addaleax mentioned this pull request May 3, 2019

Domains async_hook based implementation leaks active domain on resources maintained by internal pools #27550

Closed

Flarna mentioned this pull request May 30, 2019

[v10.x] async_hooks: avoid double-destroy HTTPParser #27986

Closed

2 tasks

basti1302 mentioned this pull request Dec 28, 2019

test: correct isReused parameter in test-http-parser #23412

Closed

2 tasks

Flarna mentioned this pull request Sep 12, 2023

The http parser is not async, why does it hook into the async_hook machinery? nodejs/performance#108

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

src: do not reuse HTTPParser #25094

src: do not reuse HTTPParser #25094

Drieger commented Dec 17, 2018 •

edited

mmarchini commented Dec 17, 2018 •

edited by Trott

mmarchini commented Dec 17, 2018

mcollina left a comment

Drieger commented Dec 17, 2018

richardlau commented Dec 17, 2018 •

edited

Flarna commented Dec 17, 2018

mscdex commented Dec 18, 2018 •

edited

AndreasMadsen left a comment

Drieger commented Dec 19, 2018

Flarna commented Dec 19, 2018

Flarna Dec 19, 2018

Flarna Jan 10, 2019

Drieger Jan 10, 2019

Drieger Jan 11, 2019

Drieger commented Jan 9, 2019

Flarna commented Jan 9, 2019

Drieger commented Jan 9, 2019

Drieger commented Jan 10, 2019

Drieger commented Jan 14, 2019

mcollina Jan 15, 2019

Drieger Jan 15, 2019

mcollina commented Jan 17, 2019

mcollina commented Jan 17, 2019

mcollina commented Jan 19, 2019

Drieger commented Mar 6, 2019

mcollina left a comment

mcollina commented Mar 7, 2019 •

edited

mmarchini commented Apr 22, 2019

mmarchini commented Apr 22, 2019

Flarna commented Apr 22, 2019

src: do not reuse HTTPParser #25094

src: do not reuse HTTPParser #25094

Conversation

Drieger commented Dec 17, 2018 • edited

Checklist

mmarchini commented Dec 17, 2018 • edited by Trott

mmarchini commented Dec 17, 2018

mcollina left a comment

Choose a reason for hiding this comment

Drieger commented Dec 17, 2018

richardlau commented Dec 17, 2018 • edited

Flarna commented Dec 17, 2018

mscdex commented Dec 18, 2018 • edited

AndreasMadsen left a comment

Choose a reason for hiding this comment

Drieger commented Dec 19, 2018

Flarna commented Dec 19, 2018

Flarna Dec 19, 2018

Choose a reason for hiding this comment

Flarna Jan 10, 2019

Choose a reason for hiding this comment

Drieger Jan 10, 2019

Choose a reason for hiding this comment

Drieger Jan 11, 2019

Choose a reason for hiding this comment

Drieger commented Jan 9, 2019

Flarna commented Jan 9, 2019

Drieger commented Jan 9, 2019

Drieger commented Jan 10, 2019

Drieger commented Jan 14, 2019

mcollina Jan 15, 2019

Choose a reason for hiding this comment

Drieger Jan 15, 2019

Choose a reason for hiding this comment

mcollina commented Jan 17, 2019

mcollina commented Jan 17, 2019

mcollina commented Jan 19, 2019

Drieger commented Mar 6, 2019

mcollina left a comment

Choose a reason for hiding this comment

mcollina commented Mar 7, 2019 • edited

mmarchini commented Apr 22, 2019

mmarchini commented Apr 22, 2019

Flarna commented Apr 22, 2019

Drieger commented Dec 17, 2018 •

edited

mmarchini commented Dec 17, 2018 •

edited by Trott

richardlau commented Dec 17, 2018 •

edited

mscdex commented Dec 18, 2018 •

edited

mcollina commented Mar 7, 2019 •

edited