Record network requests?

[elliot-nelson]

I've been using msw for nodejs unit tests and it's great. But, more recently, we were investigating using it for some larger integration tests.

What would be great is if I could run a test suite with msw enabled, but passing through all requests to the real internet; then, write out a complete set of "recorded" requests and responses that occurred, and check them in. Then I could activate msw for real, passing in the set, and the tests could run even in walled off environments with no internet access.

Is this possible today and/or a possible feature?

[kettanaito]

Hey, @elliot-nelson.

Currently, we don't have a designated API for recording outgoing requests but you can use the setupServer to introspect the traffic (specifically, the Life-cycle events API).

import fs from 'fs'
import path from 'path'
import { setupServer } from './node'

const requests = new Map()
const transactions = new Set()
const server = setupServer()

server.events.on('request:start', (request) => {
  // Record every dispatched request.
  requests.set(request.id, request)
})

// Since we're not providing any request handlers
// to the "setupServer" call, all responses will be
// bypassed (performed as-is). This will allow us to
// collect the actual responses.
server.events.on('response:bypass', (response, requestId) => {
  const request = requests.get(requestId)
  transactions.add([request, response])
})

// Add a callback to when the process exits (CTRL+C).
// Alternatively, add a callback to when your test runner
// has finished.
process.on('SIGINT', () => {
  const json = []

  for (const [request, response] of transactions.entries()) {
    json.push({ request, response })
  }

  // Write the transactions to the disk.
  fs.writeFileSync(
    path.resolve(__dirname, 'traffic.json'),
    JSON.stringify(json, null, 2),
  )
})

We are working on a library that'd make it easier to work with request handlers, including runtime traffic introspection and handlers generation (the concept is to generate handlers from various sources: runtime traffic, HAR, OpenAPI). If this feature is crucial for you, please consider raising a matter of sponsoring us in your company, becoming our first golden sponsor. That'd give us enough resources to develop the said library faster.

[kettanaito]

Giving an update on this, we're in the final stage of the implementation for a new library called Source. It will allow generating request handlers out of network archives (HAR) and Open API specification documents. This makes actual network snapshots (and specifications) to be the source of truth.

I can't give any ETAs on when it will be released but I'd expect it to release this year. Stay tuned and follow our announcements on Twitter when this happens. Thanks.

[therynamo]

@tony-scio - no plumbing on my end. I was using Polly to draw similarities to the feature request.

[zachfeldman]

Not to beat a dead horse @kettanaito but any updates on Source? Thought I'd ask since it's been a few months! It sounds awesome 😍

[kettanaito]

@zachfeldman, hey 👋 The horse is alive and kicking just in secluded stables for now. I spent some time on Source this year and I'd say it's ready to be released in its current state. That being said, my main focus goes to the MSW V2 effort so Source is unlikely to emerge anytime this year. I'm thankful for understanding about me taking time to provide quality software 🙏

[DavidOrchard]

Checking in again on Source, any progress? TIA.

[kettanaito]

@DavidOrchard, you won't miss it once there will be progress on it!
I'm buried with other work, there's a volunteer who's going to polish Source before it gets public. Let them the time they deserve.

[ipapast]

hi @kettanaito, based on this discussion here I understand that the project you were working on didn't kick off as released in the end?
does something like this (recording responses) exist nowadays?

[kettanaito]

Hi! The project is still there and I'm planning on finally making it open source this year. Quite a lot of things going on so I'm getting to them as quick as I can. I hope to release it in the combo with Source, so you can record the network and then feed it to the Source to generate handlers. That's the entire reason the recorder has been written.

We have one contributor also helping with Source so it should get the things going faster. I may also open source it sooner and then improve/add the missing pieces as we go.

[ipapast]

that's great to hear, thank you very much!