feat: Utilize NVD API

[jeremylong]

Fixes Issue #4732

Description of Change

Utilize the NVD API to retrieve the CVE data. This can also utilize the NVD API data feed that can be created using the vulnz-cli.

[jeremylong]

In order for these tests to pass we will likely need to store an NVD API key and use this during tests.

[jeremylong]

This PR is getting closer to being finalized. Left to complete:

1. Utilize API Key during regression testing in GH Actions
2. Document the new configuration.
3. Lots of testing

[aikebah]

LGTM, assuming that no tests were broken by the latest review patches

[s3nl]

I'd like to test the new method of scanning offline. I compiled the development version of Dependency Check 9 according to these steps:
https://github.com/jeremylong/DependencyCheck/tree/scratch/nvdapi#development-usage
I was able to successfully scan a .jar online with:
./cli/target/release/bin/dependency-check.sh --scan ./cli/target/release/lib/h2-2.1.214.jar

This generated the DB and created the following folder structure:

My plan is to zip the required files inside the release dictionary, upload this zip to our intranet and have the servers which need to scan without internet access unzip this file prior to scanning. A nightly update of this zip file would ensure the latest CVEs are used. The fact that these files are inside a release is a bit strange to me. Before version 9 all I needed was dependency-check-data/odc.mv.db.

Would that work? Or would I have to use vulnz to generate the DB as is suggested by the description of this PR:

This can also utilize the NVD API data feed that can be created using the vulnz-cli.

If I really have to use vulnz, then how would I post-process the generated DB and where should I put it in order for Dependency Check to use it?

Edit:
After extracting dependency-check-9.0.0-SNAPSHOT-release.zip and running
./bin/dependency-check.sh --project "Download vulnerability database" --data ./dependency-check-data --updateonly --nvdApiKey myapikey from the extracted folder I now have dependency-check-data/odc.mv.db like I was used to. Am I right that not much has changed (besides the NVD API key) in this regard and I can continue to create and distribute a zip file of dependency-check-data/odc.mv.db like I was used to with the current 8.x.x version?

[jeremylong]

Yes, you can continue to create and distribute the archive. In general, this is one of the better solutions to the offline problem.

[humblekofe]

Dear Jeremy,
first of all a million thanks to you for all your hard work for this great tool!
I'm really looking forward for this to be released, in the meantime I tried to combine the scratch/nvdapi branch with the vulnz version 5.0.1 to create an offline mirror as described in the documentation.

I observed a few things that may be caused by my stupidity (maybe I'm trying to put things together in a way that they are not supposed to?) but I have looked in the documentation and code and couldn't find a solution.
If it's not asked too much, maybe you could point me into the right direction on what I am doing wrong?
As you're certainly very busy, I can also just wait for the final release and try again :-)

1. Vulnz cache files: I'm calling the vulnz.jar directly like described in https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#creating-the-cache and the files are stored with filenames like "nvdcve-2023.json.gz" (even without the additional gzip command)
   When I configure the nvdDatafeedUrl to point to the jetty serving those files and run the check, it tries to download files named like "nvdcve--2023.json.gz" with two dashes. The reason is that the cache.properties on the server contains the prefix "nvdcve-" and NvdApiDataSource class additionally appends a dash after the prefix. If my understanding is correct, one or the other would need to be changed?
2. gzipped or not: After my attempt to remove the additional dashes in the NvdApiDataSource, the Download started but soon failed because illegal characters were encountered while downloading. As a quick try I decompressed a gz-file and placed it on the mirrorserver with the same filename "nvdcve-2023.json.gz" and the download succeeded.
   So how is this supposed to work? Should the files downloaded from the nvdDatafeedUrl be compressed? As their name is hardcoded in the NvdApiDataSource to be "...json.gz" that is what I had thought but for some reason it didn't work
3. I have then patched the NvdApiDataSource to download ".json" files and decompressed all *.json.gz on the mirror. All Downloads were successful but I ran in to an NPE afterwards... and now I'm completely confused if I'm doing the right thing here.

Thanks again and best wishes,
Felix

[jeremylong]

@humblekofe there were bugs in the nvd data feed implementation; commit 3df710e should have resolved them all.

[humblekofe]

@humblekofe there were bugs in the nvd data feed implementation; commit 3df710e should have resolved them all.

You are amazing, thank you for your quick response and fix!
I tried it out and it worked perfectly after one tiny additional modification in NvdApiDataSource.processDatafeed(...): the download URL for the *.json.gz files now contains two slashes (e.g. ".../cache//nvdcve-2005.json.gz" because there is a "/" appended in line 127-129.
If this is removed plus on the other hand added to the download of the cache.properties in row 130 (or 535 as it is required there), the cache update works.

[jeremylong]

@humblekofe what was the exact command line you used? I'm trying to understand the comment you made about the double slashes because in my testing everything is correct as is.

[humblekofe]

I'm using the maven plugin and have the following configuration:
<nvdDatafeedUrl>http://internal-server:8081/nvd-api-mirror/nvdcve-{0}.json.gz</nvdDatafeedUrl>
which results in
org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to retrieve 'http://internal-server:8081/nvd-api-mirror//nvdcve-2018.json.gz'; Error downloading file http://internal-server:8081/nvd-api-mirror//nvdcve-2018.json.gz; unable to connect.

My understanding is that in NvdApiDataSource.java

• Line 123 cuts off the trailing slash from "url" but line 128 adds it again
• Line 122 creates the pattern but including the leading slash resulting in pattern="/nvdcve-{0}.json.gz"
• both combined in Line 498, 592 etc. resulting in double slashes.
  Using nvdDataFeedUrl.substring(lio+1); in Line 122 fixes it for me.

I hope that helps.

[humblekofe]

@jeremylong : I figured out why I'm having this problem and you may not experience this:
I'm serving the cache files with a jetty which rejects URLs with double slashes (see https://stackoverflow.com/questions/74395500/jetty-returns-400-when-uri-has-2-slashes-in-a-row-while-tomcat-doesnt-care). I don't know how other web-servers behave but if I use a Tomcat, the owasp check can successfully download the files although the requested URLs contain the double forward slash.
Double slashes are usually considered as a URI violation and I would consider them as a minor bug in owasp dep-check but the workaround with using a tomcat works.

[jeremylong]

@humblekofe what I missed is that line 122 was:

pattern = nvdDataFeedUrl.substring(lio);

But should be:

pattern = nvdDataFeedUrl.substring(lio + 1);

I'll submit a PR shortly to fix this before publishing.

[jeremylong]

You can also specify the data feed url like this:

<nvdDatafeedUrl>http://internal-server:8081/nvd-api-mirror/</nvdDatafeedUrl>

[jeremylong]

@humblekofe see #6096

[humblekofe]

@jeremylong thanks a lot for the fix and also for the hint regarding the alternative configuration of nvdDatafeedUrl that will wirk with 9.0.0 as well.