Initial investigation of providing overrides in audit results per RFC #37

[iarna]

Edit: I don't know if this really the right place for this? But it's not a bug, and not a feature request -- it's discussion of predevelopment research.

Context: RFC #37 introduces the idea of allowing the audit end point to fix problems with overrides rather than just setting versions as undesirable.

To get a feel for how this will work, I wrote a proxy for the bulk audit end point that injects overrides into that. I also wrote a stand alone client that submits packages to that end point and adds overrides to the package.json based on them.

This happy path is straightforward and unsurprisingly works well.

My concern here is backwards compatibility. Running an audit using npm v7 using this proxy end point. Doing this I learned a few things:

1. The addition of overrides must not result in adding results to "vulnerable_versions" because entries there make npm try to select a new resolution that doesn't include them.
2. If you don't include "vulnerable_versions" then all versions seem to be treated as vulnerable.

Observed behavior using this end point with npm 7:

1. Running npm audit with a single an override is suggested results in a report of security issue that npm 7 does not know how to fix, but it can provide context in the result. "npm audit fix" does not do anything. This is the desired result.
2. Running npm audit on a large project with some overrides, sometimes seems to result in npm thinking it can "fix" versions that only have overrides. Running "npm audit fix" reinstalls many modules but does not seem to change any versions. This is pretty undesirable but I'm also not 100% convinced this isn't some other weird interaction happening -- I'll investigate further.

Next steps:

1. I'm going to get a public version of the proxy and client up so other folks can experiment with them. (The current version is sourced to real, but proprietary, overrides, so it requires updating.) I'll link to that here when it's up.
2. I'd like to discuss the impact of these backwards compatibility concerns on going ahead with this, and with what we might do to mitigate them.

[MylesBorins]

Hey @iarna I feel this this is a discussion that could happen in the RFCs repo so I'm transferring it there. LMK if you have any other questions or need help with next steps.

[iarna]

Sounds good, I wasn't sure where it should really live.

[iarna]

I... didn't realize there was a discussion forum attached to rfcs, actually. =D