💌📯 Updates!

[gr2m]

A while ago nock received a generous donation by Coinbase on https://opencollective.com/nock. We used some of the funds for ongoing maintenance, but most of it remained untouched.

18 days ago I suggested to my fellow maintainers @mastermatt, @paulmelnikow, and @RichardLitt that I could work onnock throughout November and December using $3,000 of the available funds.

My stated goal:

I'd go through the open issues and PRs and try to work towards making nock decomposable and easier to maintain. I hope to also use the time to setup nock for to have at least a monthly "heartbeat" of activity with minimal maintenance overhead, and to use the time I work on it to reach out to more potential sponsors, showing that things are happening and that we have a plan.

@mastermatt and @RichardLitt support my suggestion, we didn't yet hear back from @paulmelnikow

Today was my first day! I plan to mostly work on Sundays on nock. Here is a summary of what I did today:

• I updated the avatar for the maintainers team to the profile pictures of the maintainers. It's not publicly visible unfortunately, but I set it to a combination of our 4 portraits, as a reminder that we are an all white, male, US based maintainers team. I want us to do better than that in future
• Invited Richard and Matt to be maintainers of nock's Open Collective
• I re-discovered the nock maintainathon. I'll try to revive it, but probably limit it to monthly.
• I reviewed and triaged all open pull request and issues. I was able to merge/close several of them
• I uninstalled the stale bot. It closed some pull requests that I don't think should have been closed. I still think we should have some automation around closing stale issues/PRs, but only if we mark them explicitly as waiting for feedback from the author and they never respond.

Next steps

• I want to work towards decomposing nock into smaller modules. In particular I want to create
  1. @nock/intercept - hook into Node's APIs for native request interception
  2. @nock/queue - not sure about the naming yet, but basically this should hold the logic about how interceptors are added and "used"
  3. @nock/recorder - basically all of Nock Back

Notes

Most of the remaining pull requests are related to the recorder or Nock Back. The problem of the current @nock maintainers is that none of us currently use this feature. My hope is that once we I complete the decomposition of nock into smaller modules, we can also onboard new maintainers for specific aspects of nock, so people who care and depend on the recorder and Nock Back can help maintain it and have a say in how it evolves. @kibertoad offered to help already: #1494 (comment)

For reference, the pull requests are:

• [NockBack] Record "unmatched" requests only. #1235 by @guerrerocarlos
• add removeInterceptorByPredictor to make interceptor removal logics customizable #2141 by @pillsilly
• feat(scope): add filtering when loading definitions #2186 by @matthyy
• Added Update mode #2241 by @rkesters

Anyone interested in helping to make nock work nicely with http2? I don't know what the implications are right now, but we can discuss here: #1625

Next steps

• Replace Dependabot with Renovate. I did the same in several other Open Source projects. Dependabot is not a good fit for libraries, it's tailored towards applications, where each in-range update should be a new pull request. Renovate greatly reduces that noise and hence reduces maintenance overhead.
• (Re-)Familiarize myself with the code and document the code base for future me and other maintainers. I'll probably do a live-coding session at https://www.twitch.tv/gregorcodes while I look through the code next week. I'll announce it on Twitter when I do
• Once familiar with the code, I'll work on a detailed plan of how we can decompose nock into smaller modules. It's something I did with great success with for @octokit, and I'm looking forward to split up nock's code base into smaller modules and to hopefully onboard new maintainers that care about certain aspects of nock, such as the recorder.

Budget

⏳ I used up 1:57h out of the 30h budget

[mastermatt]

I'm very excited about getting some much needed love back into this library.

@gr2m we've discussed it in the past and am with you 100% on decomposing Nock into separate libs.
My one request would be that the "request matching" bits get decoupled. Probably into it's own lib. Something that's abstracted away from any of Node's implementations, especially with everything the Node HTTP team is planning on atm, and instead focuses solely on the HTTP spec stuff. I've struggled, while working on Nock, to improve the debugging/dev-experience that comes with setting up new Nocks in tests and it always seems to be other aspects of Nock's implementation that get in the way.
If you could break out what is currently there, I'd really like to take another swing at making it better.

[gr2m]

I agree, the request matching is really sub-optimal right now

[RichardLitt]

This is exciting! Thanks @gr2m for your detailed, wonderful update. Best of luck.

[peterthankgod]

This is Mind blowing development. More strength!!!!

[gr2m]

Heads up: I'll be live coding on nock today, 12:30pm US Pacific Time. You can find out details at gr2m/helpdesk#53

[gr2m]

Day 2

I started of the day by completing the new issue forms for bug reports and feature requests. When you use one of the templates, you will now see a form. And I added a link to discussions for questions:

https://github.com/nock/nock/issues/new/choose

After that I dove into the nock code. I could see remains of @paulmelnikow's work to understand the code base and amend it with useful comments. That was tremendously helpful. I recorded my attempts to asses the current state of the nock code base.

1. https://www.twitch.tv/videos/1205654930
2. https://www.twitch.tv/videos/1205809190

No need to watch the 5h, it's really not that interesting :) But I'll try to make it a regular thing. Feel free to pop in & out on Sundays, around noon - 5pm US pacific time.

I started the work on a @nock/intercept module. I got the request interceptions bit, but have yet to full understand mocking the responses in a way that request libraries such as got work as expected. I started a work-in-progress pull request.

The most relevant learnings from this session

1. nock is replacing 5 native APIs
   • http.ClientRequest which is used by both node's http and https modules
   • http.request and http.get
   • https.request and https.get
2. In today's nock codebase, the logic of intercepting requests, mocking responses, the "mocks queue" and global flags to turn features on/off using environment variables is heavily intertwined, which makes it very hard to understand.
   • In lib/intercept.js is today's code that replaces the native http.ClientRequest, which later is also called from the overwrites of {http,https}.{get,request}. On line 277 the interceptors are loaded for the current request options, and then passed to theInterceptedRequestRouter on line 289. Once instantiated, the properties of overrider are applied to the instance of the overridden http.ClientRequest, which makes it compatible enough to be consumable by request libraries such as got.
   • What I'm trying to do is remove the logic that matches request options to defined mocks out of the intercept logic. I am trying to add a single callback function as a way to hook into the moment when request options are parsed and normalized and a mock response has to be returned. That way @nock/intercept could be used standalone with custom logic to match a request to responses. I'm not sure if a single callback will suffice or if a more sophisticated set of hooks need to be exposed, that's what I plan to figure out during my next session
   • Once I get the new implementation @nock/intercept working in the local modules/intercept folder, I will try to use it in code's nock base, until all tests pass. I expect this to keep me busy for the next 2-3 weeks. But once achieved, I can create the first external module, use it as dependency, and move on to the next bit, which will probably be defining of mocks and matching with request options.

Let me know if you have any questions :) See you next week

Budget

⏳ I used up 6:52h out of the 30h budget

[gr2m]

I worked some more on the intercept module the past two days and managed to mock my very first request!

Besides mocking http.ClientRequest, nock is creating a custom http.IncomingMessage based on the matched interceptor (mock).

What I think we should do is pass a callback function to the new intercept() method, which will be called with the request options and return either a response object (http.IncomingMessage instance) or nothing, in which case the request is sent through without any interception. That way we can dynamicall

• return a mocked response
• throw a useful error in case no mocked response could be found
• let the request go through

One last thing that I'd like to also make easily possible is to let the request through, but also be able to record the response. I think with that we would cover all use cases required by nock, and we can move the logic of matching requests with a list of mocks (interceptors) out of the intercept logic.

This is more complicated than I anticipated. I've now used up roughly a third of the budget, but I think I do have a pretty good understand of how nock intercepts Node requests/responses by overwriting selective native APIs and how we can decouple that logic from the rest and move it into a separate @nock/intercept module.

My goal for the next 10h or so is to finish up the intercept API as described above, then refactor nock's current code to use a local module and make all existing tests pass.

After that I think it would make sense to create the separate repository and module, install it as dependency, and make sure the separate repository has its own solid test suite and documentation. Once we have that we can use it to demonstrate what we have done with a focused sprint of paid work, lay out what the other modules are that we want to extra from nock, and use both as a means of fundraising for more sponsors.

Budget

⏳ I used up 9:55h out of the 30h budget

[gr2m]

Day 3

Before starting the coding session on nock I updated our Open Collective at https://opencollective.com/nock and submitted @nock for GitHub Sponsors. Everyone, especially companies, will soon be able to sponsor nock directly on GitHub

After that I worked on the nock refactoring as layed out in this issue: gr2m/helpdesk#54

I streamed my refactoring work again, you can see the videos at

1. https://www.twitch.tv/videos/1212196617
2. https://www.twitch.tv/videos/1212336363

I accomplished a few things today:

1. Intercept a request using a callback passed to intercept. See examples/hello-world.js
2. Do not intercept a request based on request options. See examples/conditional-intercept.js
3. Use the new intercept module within nock and make the first integration test pass. See tests/test_intercept.js

I prepared another issue for next week: gr2m/helpdesk#55. I'm pretty sure I'll work the entire show through the existing tests and make as many of the pass again as possible. I'll try to resist any refactoring until all tests are passing again.

Once all tests are passing again, we accomplished the most minimal goal of refactoring nock's codebase to separate the logic to overwrite native Node APIs in order to intercept request from the rest of the code, which defines mocks and matches them with a request. In case I run out of hours, that is the absolute minimal state that I'd leave the code in. Doing refactorings and moving the internal intercept module is the fun part, I can do that in after hours when I have more time again, or someone else can finish up that work.

But I am still optimistic that I manage to end up with a separate nock/intercept repository, that is fully documented, tested, with it's on @nock/intercept module that is used by nock internally, but that could be used standalone for simple use cases.

Budget

⏳ I used up 15:09h out of the 30h budget

[gr2m]

Day 4

I continued working on the failing tests and got about 50% of the tests to pass. I checked of the files that are now passing in this issue: gr2m/helpdesk#55

I streamed my refactoring work again, you can see the video at https://www.twitch.tv/videos/1218606526

One thing that become clear to me today is that we should name the intercept module @nock/intercept-node-http. That way we could other @nock/intercept-* modules in future or encourage the community to build their own, e.g. for intercepting fetch requests in the browser or intercepting the lowerlevel net module instead of http.

Another thing I cleaned up today was the use of the proto variable, which I've also seen @paulmelnikow to be confused by in some places based on his comments. It's just protocol now.

I prepared an issue for the next week, see you then: gr2m/helpdesk#56

Budget

⏳ I used up 18:36h out of the 30h budget

[gr2m]

Day 5

I took a fun detour today. I learned about @moll's mitm when I saw a nock issue back from 2014 where Andri mentioned the library.

I definitely recommend everyone to check it out, it's very elegant! I'm not sure if it covers all use cases that nock requires, but I felt like we could learn some tricks or even adapt mitm as an alternative to our own @nock/intercept module, or even better: combine the two into something new and create the best of both worlds.

Two things that stand out in mitm

1. mitm does not create it's own version of http.ClientRequest. Instead it overwrites ClientRequest.prototype.onSocket which is called synchronously in the ClientRequest constructor.
2. It event supports mocking net requests. That way it can be used for mocking request from libraries that do not use Node's http/https modules at all, such as https://github.com/nodejs/undici.

I ended up rewriting Andri's module as a native ES Module just for the fun of it:
https://github.com/gr2m/mitm-esm

You can find some notes and link to the recording of this session at gr2m/helpdesk#57

Budget

⏳ This one's on me. Budget status remains at 18:36h out of 30h

[jimmywarting]

Would love to have a thing such as FetchEvent.respondWith()

Have been thinking of such thing for node-fetch

[gr2m]

nice! I could totally imagine to build such an interface on top of the lower-level libraries that intercept the requests. A higher level API could work like a service worker that would use native APIs to either intercept a request or pass it through, and the test APIs could work the same in Node.js and browsers, only the runtime for the test would differ

[jimmywarting]

it would have to support a ExtendableEvent that can wait for a promise to resolve... somehow

[gr2m]

Day 6

I got back to working on nock this week and got a lot more tests to pass, albeit not all of them yet:
gr2m/helpdesk#58

I streamed my refactoring work again, you can see the video at https://www.twitch.tv/videos/1231751326

I ran into a few breaking changes

• Before, the intercept logic occurred directly in the overridden http.ClientRequest constructor. Now, the intercept logic is only called if an actual request is being sent.

  Before

  // called `errorHandler` immediately, before request is even sent
  new http.ClientRequest({ port: 12345, path: '/' }).on('error', errorHandler)

  After

  // called `errorHandler` not called until request is sent.
  const request = new http.ClientRequest({ port: 12345, path: '/' }).on(
    'error',
    errorHandler
  )
  request.end()

• When a request is not intercepted, we call the interal request.nockSendRealRequest() method provided by the new intercept module. When that happens, the real request is sent out, but the orginal http.{get,request} methods are not called. The propor fix to this breaking change is to not override http.{get,request} and http.ClientRequest in the first place and instead hook into the http.ClientRequest constructor, as mitm does.

After working on mitm past week and seeing this week how much headache is caused by the fact that we have overwritten the entire http.ClientRequest and http.{get,request} methods instead of just hooking into the http.ClientRequest constructor makes me really want to throw it all away 🤣 It's always greener on the other side. I'm not going to do that of course. I'll try to continue until all the existing tests pass and we have the intercept logic fully isolated. Only once that is done, I'll see if we can replace it with something simple than we currently have.

Budget

⏳ I used up 22:46h out of the 30h budget

[gr2m]

Worked some more, only two tests files remaining, both related to the recorder.

• tests/test_back_filters.js
• tests/test_recorder.js

Budget

⏳ I used up 25:23h out of the 30h budget

[gr2m]

Day 7

I managed to make the remaining two tests pass quite quickly at the beginning of today's show
gr2m/helpdesk#59

However, nock's recorder logic was still using the old intercept logic, not my new module. So I spent about 4h of refactoring the current recorder to make it use the new module and work towards making the same tests pass, again.

I think I got the intercept module at a point where it can be used to record the real request and response instead of intercepting, but I did not yet manage to make all the tests pass.

Unfortunately I used up all the available budget. We have extra budget of ~6h in our collective, but I want to keep that to fund the review of the massive change for one of the other maintainers, and leave some extra hours for follow up items in case anything comes up that cannot be easily fixed.

I suggest that I keep track of time, I assume I'll need another 6-12h. If we find another sponsor that we can fund that work retroactively. If not, 🤷🏼 it's on me.

Budget

🚨⏳ I used up 30:43h out of the 30h budget

[RichardLitt]

Good work. :)

[gr2m]

After some frustration with the way nock's request recording feature works, I had a look around last night to see what else is out there. As far as I can tell, there is no dedicated module for in-process http request/response lifecycle recording. So of course I had to build my own: https://github.com/gr2m/http-recorder#readme

It's not thoroughly tested, I hope it will live up to what we need in nock. The implementation is so simple, it warms my maintainer's heart ❤️

[gr2m]

Day 8

All tests are passing, and all code except a single branch is covered

The remaining branch was cover by a test that I had to skip for now, in which we call nock.cleanAll() on the socket event of the request. I think I can recover that behavior once I move the intercept logic to https://github.com/moll/node-mitm or something derived from it, because it hooks into the connect event of the net module which comes before the http module is even used. In the connect event it decides whether to intercept a request or not, so we could load the current interceptors at this point and use them when mocking a request later, even if the interceptors have been all deleted at the time the request emits the socket event.

One breaking change that is that a request has to be send before nock can do anything. In the current code, nock runs the intercept logic in the http.ClientRequest constructor. My new code runs it after the socket event was emitted. Mitm does both so it might be possible to recover the current behavior but I'm not 100% sure.

There are two remaining failing tests that I've currently skipped about the http.{get,request} method that I'm sure will work again once we change the intercept logic to Mitm, so I consider them TBD.

I'm not sure if it makes sense to move forward with the current code where the intercept logic is already isolated, but more complicated than it has to be, or if we should try to use the Mitm-inspired interception logic instead.

I'll be very busy the next 3 weeks, but I might have a look into it in-between things.

Budget

🚨⏳ I used up 35:30h out of the 30h budget

[gr2m]

Thanks to festive procrastination of chores I built two more modules:

• https://github.com/gr2m/node-http-interceptor
• https://github.com/gr2m/node-net-interceptor

I'll try to use https://github.com/gr2m/node-http-interceptor & https://github.com/gr2m/node-http-recorder as a replacement for what I currently extracted from nock's code base and see if it's feasible or not.

[RichardLitt]

A general overview might be nice! You covered so much ground. Thanks for this work, Gregor.

[gr2m]

I'm keeping an up-to-date summary at #2252. I don't want to extract the intercept logic into its own module yet, I want to explore if we can use the @gr2m/node-{http,net}-{interceptor,recorder} packages first. I'll not let it go stale, if it's too much work I'll work through the remaining todos with the current implementation and leave the use of the new packages to another refactoring

[gr2m]

I'll get back to it by Feb 13th the latest. I hope to find time in one of the earlier Sundays to at least look at it, but will not do a live coding show before Feb 13th

[peterthankgod]

I see the program you have worked on and it is amazing, it is the capacity building for others to key in and step up the best in them.

[gr2m]

Hey just a quick update: I had to take a break from my work on refactoring nock due to several significant life events. I had to step back from pretty much all my Open Source work. Things are getting better now and I hope to find time for nock soon. I'm very much looking forward to it, it's a great creative output.