Feature/reduce commit frequency

[attilakarpatiorion]

Description

The change reduces the number of database commits when parsing CVEs to improve performance in cases when latency to the database matters. 

The changes provide improvements in two ways: batching 500 updates before committing changes, and enclosing multiple non-transactional field updates into a single transaction.

Addressed Issue

Link: #2895

Compared to a local in-instance DB the performance with RDS PostgreSQL dropped significantly.

Performance testing has shown that with AWS RDS PostgreSQL on db.t3.medium and the api server on t3.xlarge the initial load time for the 2023 JSON was 6 minutes 20 seconds before the changes, and 3 minutes 33 seconds after the changes. For the 2022 JSON the load time has changed from 11 minutes 37 seconds to 7 minutes 58 seconds.

As a comparison, with a local DB instance the load time for the 2022 data was 3 minutes 46 seconds.

Additional Details

The code changes don't affect the behaviour of the application, just the performance.
To verify if that's the case it's sufficient to run existing test cases, no new test cases are needed.

Re-running the test cases were successful when setting TZ=UTC, with NZST time zone SnykAnalysisTaskTest.testAnalyzeWithRateLimiting:309 failed.

Output of 'mvn clean verify -P enhance' was

INFO] Results:
[INFO]
[WARNING] Tests run: 1173, Failures: 0, Errors: 0, Skipped: 2

Further improvements are possible by hiding latency by increasing the number of threads which do read only queries, and distributing those queries to read only replicas.

Checklist

• I have read and understand the contributing guidelines
• This PR fixes a defect, and I have provided tests to verify that the fix is effective
• This PR implements an enhancement, and I have provided tests to verify that it works as intended
• This PR introduces changes to the database model, and I have added corresponding update logic
• This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

[sonatype-lift]

Sonatype Lift is retiring

Sonatype Lift will be retiring on Sep 12, 2023, with its analysis stopping on Aug 12, 2023. We understand that this news may come as a disappointment, and Sonatype is committed to helping you transition off it seamlessly. If you’d like to retain your data, please export your issues from the web console.
We are extremely grateful and thank you for your support over the years.

📖 Read about the impacts and timeline

[sonatype-lift]

🛠 Lift Auto-fix

Some of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.¹

# Download the patch
curl https://lift.sonatype.com/api/patch/github.com/DependencyTrack/dependency-track/2896.diff -o lift-autofixes.diff

# Apply the patch with git
git apply lift-autofixes.diff

# Review the changes
git diff

Want it all in a single command? Open a terminal in your project's directory and copy and paste the following command:

curl https://lift.sonatype.com/api/patch/github.com/DependencyTrack/dependency-track/2896.diff | git apply

Once you're satisfied, commit and push your changes in your project.

Footnotes

1. You can preview the patch by opening the patch URL in the browser. ↩