Return semantic exit code for audit

[antrew]

Summary

Resolve #6748

yarn audit currently returns the number of found vulnerabilities. This can lead to an overflow and a false negative. E.g., if the number of vulnerable dependencies is equal to 256, the exit code will be equal 0. Furthermore, it is not very practical, as this number does not indicate the severity level of found vulnerabilities.

This pull request modifies this behavior by returning a number that is a sum of found severities:

• 1 for INFO
• 2 for LOW
• 4 for MODERATE
• 8 for HIGH
• 16 for CRITICAL

That is, if only INFO and MODERATE vulnerabilities were found, then the exit code will be 1+4 = 5.

With this change it would be possible to check for particular severity levels. E.g., if someone wants to know, if there are HIGH and CRITICAL dependencies, then one could check for exit code >= 8.

Test plan

Current behavior:

$ yarn audit
9 vulnerabilities found - Packages audited: 31177
Severity: 9 Low
Done in 1.94s.
$ echo $? 
9

New behavior:

$ yarn-local audit
...
9 vulnerabilities found - Packages audited: 31177
Severity: 9 Low
Done in 1.69s.
$ echo $?
2

[arcanis]

Technically it should be a major bump, but I wouldn't be opposed to put that into the next minor. What do you think, @rally25rs?

[antrew]

Hi @arcanis,

I don't think anybody relies on the current implementation, since it doesn't really matter if you have 1, 10 or 99 vulnerabilities. In my opinion, the only thing that makes sense with the current implementation, is to check if the exit status is zero (no vulnerabilities), or non-zero (there are vulnerabilities).
In that sense the proposed implementation is backwards compatible.

Anyway, it seems like everyone is building some wrappers around the current implementation in their CI pipelines to find out if they have vulnerabilities of certain levels. Ours looks like this at the moment and has to completely ignore the exit code of yarn audit:

yarn audit || true                                  
if [[ `yarn audit | grep -oiP '(critical|high)'` ]] ; then     
  echo 'Critical vulnerabilities found'             
  exit 1                                            
else                                                
  exit 0                                            
fi                                                  

[rally25rs]

I like this change. Good idea! 👍

[rally25rs]

I added the appropriate docs website change: yarnpkg/website#894