[network-plugin] Feat: Capture network events #1105

jlalmes · 2023-01-31T19:22:07Z

Closes #552

This implementation records fetch and XMLHttpRequest by patching their object & methods. We record document navigation using PerformanceNavigationTiming and we use PerformanceResourceTiming for recording everything else (script, img, link etc.) via PerformanceObserver API.

Recording

type NetworkRequest = {
  url: string;
  method?: string;
  initiatorType: InitiatorType;
  status?: number;
  startTime: number;
  endTime: number;
  requestHeaders?: Headers;
  requestBody?: string | null;
  responseHeaders?: Headers;
  responseBody?: string | null;
};

rrweb.record({
  ...,
  plugins: [
    getRecordNetworkPlugin({
      initiatorTypes: ['fetch', 'xmlhttprequest'], // omit to record all types
      ignoreRequestFn: (request: NetworkRequest) => false
      recordHeaders: true, // defaults to false
      recordBody: true, // defaults to false
      recordInitialRequests: true, // defaults to false
    })
  ]
})

initiatorTypes is an array specifying which request types to record
ignoreRequestFn is a function that allow you to block recording an event for a request (you don't want to create a loop recording the request you send to store rrweb events on your server)
recordHeaders will record an request/response headers for fetch and xmlhttprequest
recordBody will record an request/response bodies for fetch and xmlhttprequest
recordInitialRequests will record an event for all requests prior to rrweb.record() being called

Replaying

new rrweb.Replayer(events, {
  ...,
  plugins: [
    getReplayNetworkPlugin({
      onNetworkData: (data: { requests: NetworkRequest[], isInitial?: boolean }) => { // required
        // do something
      }, 
    })
  ]
})

You must supply your own onNetworkData function.

To-do

Tests

packages/rrweb/src/plugins/network/record/index.ts

YunFeng0817 · 2023-02-08T05:36:16Z

Thanks for applying all review suggestions. Although there are still a few small points that need to tweak, it's very close to the final version.

FYI here is a TODO list:

improve the content type filter
test cases
Chinese docs (I will add them later)

I'm so sorry that, for the release of 2.0.0-alpha.5, I also don't have a specific date to tell. These days other team members are all busy with their work and some important pull requests for bug fixes are delayed.

YunFeng0817

With more tests, I think this pull request is good to go

Juice10 · 2023-02-09T09:09:16Z

This looks great! I agree with @Mark-Fenng, if we have some more tests this will be ready to merge. Also I'm happy to spend some time working on the 2.0.0-alpha.5 release soon

docs/recipes/network.md

Co-authored-by: Ben White <ben@benjackwhite.co.uk>

changeset-bot · 2023-02-20T10:19:50Z

⚠️ No Changeset found

Latest commit: 8863576

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

benjackwhite · 2023-04-20T11:36:58Z

@jlalmes need some help getting this across the finish line? This would be such a good improvement to the custom solution we have for network recording and things here look pretty solid, just needing some good tests it seems.

yaoyuanrong · 2023-05-09T03:43:59Z

Is this plugin released?

yaoyuanrong · 2023-05-09T03:46:45Z

I really need this feature。I think this is a very powerful feature @jlalmes

benjackwhite

Saw this is starting to get reviewed and wanted to throw in a thought regarding ignoreRequestFn to see if we can't make it more flexible, as well as more inline with the text and input style approach.

benjackwhite · 2023-10-19T18:23:55Z

packages/rrweb/src/plugins/network/record/index.ts

+    const requests = data.requests.filter(
+      (request) => !networkOptions.ignoreRequestFn(request),
+    );


It would be great if this was both filtering and cleansing.

The most typical case for privacy controls is:
"I want to filter out requests to this place"
"I wan't to ensure certain headers aren't included"
"I want to filter out PII from the response bodies".

This would all doable with a slight modification of the function:

Suggested change

const requests = data.requests.filter(

(request) => !networkOptions.ignoreRequestFn(request),

);

const requests = data.requests.reduce(

(list, request) => {

const maskedRequest = networkOptions.maskRequestFn(request)

return maskedRequest ? [...list, maskedRequest] : list

}, []

);

Which then allows the function to be something like

maskRequestFn: (request: NetworkRequest): NetworkRequest | undefined => { if (request.url.contains("rrweb-collector-api.com") { return } delete request.requestHeaders["Authorization"] request.responseBody = maskTextFn(request.responseBody) return request }

In theory this could be added in a follow up PR as well. Happy to do that if that is the decision.

pauldambra · 2023-11-04T10:20:08Z

packages/rrweb/src/plugins/network/record/index.ts

+  if (attempt > 10) {
+    throw new Error('Cannot find performance entry');
+  }
+  const urlPerformanceEntries = win.performance.getEntriesByName(


If I'm understanding correctly I think we'd need to make this optional...

If people already have their own code (or another dependency) tracking performance entries they might be calling https://developer.mozilla.org/en-US/docs/Web/API/Performance/clearResourceTimings already so we'd have a race.

Since this usage comes after the response if someone is clearing the buffer as soon as they're able to then I guess that this code would lose the race.

I think it's cheap enough to have the observer running no matter what that you could have recordRequestPerformance as a config option and return early here so the signature becomes Promise<PerformanceRequestTiming | null>

My mistake - 🤯 I've just read back through this and seen that the thrown error (and any other error here) are captured and swallowed where the returned promise is handled.

My preference would be to return null so that the flow is more clear but at least the error here doesn't bubble out

jasonfill · 2023-11-14T21:09:20Z

I would love this feature! Just throwing in a 👍 for this.

pauldambra · 2023-11-22T16:18:31Z

packages/rrweb/src/plugins/network/record/index.ts

+type NetworkRequest = {
+  url: string;
+  method?: string;
+  initiatorType: InitiatorType;
+  status?: number;
+  startTime: number;
+  endTime: number;
+  requestHeaders?: Headers;
+  requestBody?: Body;
+  responseHeaders?: Headers;
+  responseBody?: Body;
+};


I was experimenting with this again...

We already capture Network timing and the PerformanceEntry spec uses name where this NetworkRequest timing uses url

It is more natural for the typing here to be something like

export type CapturedNetworkRequest = Omit<PerformanceEntry, 'toJSON'> & { method?: string initiatorType?: InitiatorType status?: number startTime?: number endTime?: number requestHeaders?: Headers requestBody?: Body responseHeaders?: Headers responseBody?: Body // was this captured before fetch/xhr could have been wrapped isInitial?: boolean }

That way you can extend this to capture PerformanceEntrys and/or headers and/or body and only use one type

pathik-appachhi · 2024-01-01T21:17:37Z

Hello,

I have recently started using rrrweb-io via the cdn . It would be very helpful to know when this plugin will be available on the CDN link.

ShayMalchi · 2024-01-08T09:12:06Z

@jlalmes @YunFeng0817 Seems like this PR was approved long ago, can you please share what is the current blocker for merging? Many people in this thread are interested in it, and seems like most suggestions could be handled in followup PRs. Thank you!

pauldambra · 2024-01-08T09:39:22Z

I'll just re-up this comment #1105 (comment)

It's easier for me if we have name not URL. Since the performance observer uses name and emits more than just things with URLs.

pauldambra · 2024-02-02T11:32:57Z

We have network payload capture in prod now based on this PR - thanks so much @jlalmes 🙌

You can see the code here https://github.com/PostHog/posthog-js/blob/main/src/loader-recorder-v2.ts

We couldn't use it quite as it is... and I've had to make a couple of bugfixes since releasing it - mostly around processing bodies on unexpected types of network requests

My plan is to let it settle in our product for a week or two. And then I'll open a PR based on this one that would let us inject the things we'd varied - which might also be things other people would want to vary...

It won't be massively different - and of course the version that works for us might not be preferable generally

sharan-ravi · 2024-02-20T09:30:47Z

@jlalmes @pauldambra
Thank you so much for this!

I was running into an issue with some of my requests not being recorded with this plugin, after digging deeper, turns out in case of certain requests, if the server doesn't return a Timing-Allow-Origin response header, we wouldn't be able to track the performance metrics. Given this plugin throws an error when performance metrics are not available, these requests were not being recorded.

While the performance metrics are not available, Other data on these requests are still available, so in my case I had to do something like this to still record these requests.

getRequestPerformanceEntry(win, 'fetch', req.url, after, before)
                    .then((entry) => {
                        if (_isNull(entry)) {
                            const requests = prepareRequestWithoutPerformance(req, networkRequest)
                            cb({ requests })
                        } else {
                            const requests = prepareRequest(entry, req.method, res?.status, networkRequest)
                            cb({ requests })
                        } })

function prepareRequestWithoutPerformance(
    req: Request,
    networkRequest: Partial<CapturedNetworkRequest>,
): CapturedNetworkRequest[] {

    const reqJson = {
        url: req.url,
        method: req.method,
        requestHeaders: networkRequest.requestHeaders,
        requestBody: networkRequest.requestBody,
        responseHeaders: networkRequest.responseHeaders,
        responseBody: networkRequest.responseBody,
    }

    return [reqJson]
}

Is this expected?

Juice10 · 2024-04-18T09:40:21Z

My plan is to let it settle in our product for a week or two. And then I'll open a PR based on this one that would let us inject the things we'd varied - which might also be things other people would want to vary...

It won't be massively different - and of course the version that works for us might not be preferable generally

@pauldambra would you still be interested to open a PR based on the version you have running in production?

FYI: #1033 just got finished which moves the all of the plugins to the packages/plugins/... directory. Might be worth considering creating the PR based on that as well

akhsri · 2024-05-02T11:06:27Z

Hi @jlalmes , When can we see this in rrweb package? it would be very powerful and helpful feature!

pauldambra · 2024-05-09T10:07:31Z

@Juice10 definitely... our implementation has been pretty stable for a while now... it's just time that's stopping me 🙈

I think the "tricky" thing will be separating our config/mapping needs out of the plugin but that's pretty mechanical non-ground-breaking stuff

record xhr

7c5f909

jlalmes changed the title ~~record xhr~~ [network-plugin] Feat: Add network recording plugin Jan 31, 2023

jlalmes changed the title ~~[network-plugin] Feat: Add network recording plugin~~ [network-plugin] Feat: Capture network events Jan 31, 2023

use performance

ada77af

jlalmes marked this pull request as ready for review February 1, 2023 13:48

jlalmes added 2 commits February 1, 2023 15:25

emit less events & isInitial flag

2b23db5

stub xhr & fetch observers

dafbdd4

jlalmes marked this pull request as draft February 1, 2023 15:29

jlalmes added 11 commits February 1, 2023 15:54

stub req headers & body

6e9377a

small type refactor

27f4360

add NavigationTiming

7fa75b5

rename props

52c58b3

implement fetch

05a45df

use stringify

863717d

refactor options

e693366

add requestMethod

78b338f

rename replay prop

d656ac7

add ignoreRequestFn

1d77b6c

rn initiatorTypes

f3d85e2

jlalmes force-pushed the feat/network-plugin branch from b366ff1 to f3d85e2 Compare February 1, 2023 19:02

jlalmes added 10 commits February 1, 2023 19:07

fix build and rename prop

a503ca4

bug fix

ca21bab

move win.perf check up

70395af

better last perf entry

0e08bc7

implement xhr

9d33b47

better xhr events implementation

a174964

skip empty request callbacks

7f9ba0d

by url

ad966bc

linting

0ae1f06

fix xhr

147aba6

YunFeng0817 reviewed Feb 8, 2023

View reviewed changes

packages/rrweb/src/plugins/network/record/index.ts Outdated Show resolved Hide resolved

jlalmes added 2 commits February 8, 2023 09:56

better body typing

0278113

better recordBody

6804e4b

YunFeng0817 approved these changes Feb 9, 2023

View reviewed changes

benjackwhite reviewed Feb 20, 2023

View reviewed changes

docs/recipes/network.md Outdated Show resolved Hide resolved

Update docs/recipes/network.md

8863576

Co-authored-by: Ben White <ben@benjackwhite.co.uk>

benjackwhite mentioned this pull request Apr 20, 2023

Network Recording: Capture network request and response info PostHog/posthog#13450

Closed

antunesgabriel approved these changes Oct 19, 2023

View reviewed changes

benjackwhite suggested changes Oct 19, 2023

View reviewed changes

benjackwhite mentioned this pull request Oct 25, 2023

Sprint October 30th - November 10th PostHog/posthog#18174

Closed

pauldambra reviewed Nov 4, 2023

View reviewed changes

pauldambra reviewed Nov 22, 2023

View reviewed changes

ShayMalchi mentioned this pull request Jan 9, 2024

[network-plugin] Feat: Capture network events #1393

Closed

Juice10 assigned Juice10 and Yuyz0112 Mar 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[network-plugin] Feat: Capture network events #1105

[network-plugin] Feat: Capture network events #1105

jlalmes commented Jan 31, 2023 •

edited

YunFeng0817 commented Feb 8, 2023 •

edited

YunFeng0817 left a comment

Juice10 commented Feb 9, 2023

changeset-bot bot commented Feb 20, 2023

benjackwhite commented Apr 20, 2023

yaoyuanrong commented May 9, 2023

yaoyuanrong commented May 9, 2023 •

edited

benjackwhite left a comment

benjackwhite Oct 19, 2023

benjackwhite Oct 19, 2023

pauldambra Nov 4, 2023 •

edited

pauldambra Nov 12, 2023

jasonfill commented Nov 14, 2023

pauldambra Nov 22, 2023

pathik-appachhi commented Jan 1, 2024

ShayMalchi commented Jan 8, 2024

pauldambra commented Jan 8, 2024

pauldambra commented Feb 2, 2024

sharan-ravi commented Feb 20, 2024 •

edited

Juice10 commented Apr 18, 2024 •

edited

akhsri commented May 2, 2024

pauldambra commented May 9, 2024

-    const requests = data.requests.filter(
-      (request) => !networkOptions.ignoreRequestFn(request),
-    );
+    const requests = data.requests.reduce(
+      (list, request) => {
+        const maskedRequest = networkOptions.maskRequestFn(request)
+        return maskedRequest ? [...list, maskedRequest] : list
+      }, []
+    );

[network-plugin] Feat: Capture network events #1105

Are you sure you want to change the base?

[network-plugin] Feat: Capture network events #1105

Conversation

jlalmes commented Jan 31, 2023 • edited

Recording

Replaying

To-do

YunFeng0817 commented Feb 8, 2023 • edited

YunFeng0817 left a comment

Choose a reason for hiding this comment

Juice10 commented Feb 9, 2023

changeset-bot bot commented Feb 20, 2023

⚠️ No Changeset found

benjackwhite commented Apr 20, 2023

yaoyuanrong commented May 9, 2023

yaoyuanrong commented May 9, 2023 • edited

benjackwhite left a comment

Choose a reason for hiding this comment

benjackwhite Oct 19, 2023

Choose a reason for hiding this comment

benjackwhite Oct 19, 2023

Choose a reason for hiding this comment

pauldambra Nov 4, 2023 • edited

Choose a reason for hiding this comment

pauldambra Nov 12, 2023

Choose a reason for hiding this comment

jasonfill commented Nov 14, 2023

pauldambra Nov 22, 2023

Choose a reason for hiding this comment

pathik-appachhi commented Jan 1, 2024

ShayMalchi commented Jan 8, 2024

pauldambra commented Jan 8, 2024

pauldambra commented Feb 2, 2024

sharan-ravi commented Feb 20, 2024 • edited

Juice10 commented Apr 18, 2024 • edited

akhsri commented May 2, 2024

pauldambra commented May 9, 2024

jlalmes commented Jan 31, 2023 •

edited

YunFeng0817 commented Feb 8, 2023 •

edited

yaoyuanrong commented May 9, 2023 •

edited

pauldambra Nov 4, 2023 •

edited

sharan-ravi commented Feb 20, 2024 •

edited

Juice10 commented Apr 18, 2024 •

edited