Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement filtering by packages through the config #944

Merged
merged 8 commits into from
Jun 6, 2024

Conversation

josieang
Copy link
Collaborator

@josieang josieang commented Apr 26, 2024

I'd like feedback on the config yaml schema, the filter message and it's behaviour if the version is empty (it filters any version of that package).

This is in response to #814

pkg/config/config.go Outdated Show resolved Hide resolved
pkg/config/config.go Outdated Show resolved Hide resolved
pkg/osvscanner/osvscanner.go Show resolved Hide resolved
@josieang
Copy link
Collaborator Author

I think I need a better way to filter based on semver parts. I know think deps.dev/util/semver does this, I will come back to it once I'm back from leave.

Copy link
Collaborator

@another-rex another-rex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Main comment is that I think we can consolidate the two overrides into one, so they can share the package matching logic.

LoadPath string `toml:"LoadPath"`
GoVersionOverride string `toml:"GoVersionOverride"`
IgnoredVulns []IgnoreEntry `toml:"IgnoredVulns"`
IgnoredPackageVersions []IgnorePackageVersionEntry `toml:"IgnoredPackageVersions"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like ignore package versions and override package versions both could have the same "package matching" logic.

Can we consolidate them into the same field, and just have a bool ignore which if true ignores the package entirely, otherwise an override field where someone can override the license.

E.g.

[[Package]]
name = "pkg-name"
exactVersion = "1.0.0"
ecosystem = "Go"
ignore = false # if true ignores the package entirely
licenseOverride = ["MIT", "0BSD"]
# In the future, if someone requests we can also add a versionOverride field here, or other overrides
reason = "abc"
# major ...
# minor ...
# ...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

version (Which I think we should exactVersion to) is optional right?

Additionally, would it make sense to make license a structured field of some sort instead?

I.e.

[[Package]]
[[Package.license]]
override = ["MIT"]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me! done.

pkg/config/config.go Outdated Show resolved Hide resolved
pkg/osvscanner/osvscanner.go Show resolved Hide resolved
Allowlist: []models.License{models.License("MIT"), models.License("0BSD")},
},
},
Results: []models.PackageSource{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For maintainability, can we just use the snapshot library here to store/match the results?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be done but check my code just in case I'm not understanding the snapshot library correctly

pkg/config/config.go Outdated Show resolved Hide resolved
pkg/config/config.go Outdated Show resolved Hide resolved
LoadPath string `toml:"LoadPath"`
GoVersionOverride string `toml:"GoVersionOverride"`
IgnoredVulns []IgnoreEntry `toml:"IgnoredVulns"`
IgnoredPackageVersions []IgnorePackageVersionEntry `toml:"IgnoredPackageVersions"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

version (Which I think we should exactVersion to) is optional right?

Additionally, would it make sense to make license a structured field of some sort instead?

I.e.

[[Package]]
[[Package.license]]
override = ["MIT"]

pkg/config/config.go Outdated Show resolved Hide resolved
@codecov-commenter
Copy link

codecov-commenter commented May 7, 2024

Codecov Report

Attention: Patch coverage is 81.96721% with 11 lines in your changes missing coverage. Please review.

Project coverage is 65.18%. Comparing base (8fd553a) to head (78fe74d).
Report is 1 commits behind head on main.

Files Patch % Lines
pkg/osvscanner/osvscanner.go 8.33% 10 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #944      +/-   ##
==========================================
+ Coverage   65.14%   65.18%   +0.04%     
==========================================
  Files         149      149              
  Lines       12338    12384      +46     
==========================================
+ Hits         8037     8072      +35     
- Misses       3849     3859      +10     
- Partials      452      453       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@josieang
Copy link
Collaborator Author

josieang commented May 7, 2024

Was I supposed to provide a token?

https://github.com/google/osv-scanner/actions/runs/8978083643/job/24657957118

Error: Codecov token not found. Please provide Codecov token with -t flag.
Error: Codecov: Failed to properly create commit: The process '/Users/runner/work/_actions/codecov/codecov-action/5ecb98a3c6b[74](https://github.com/google/osv-scanner/actions/runs/8978083643/job/24657957118#step:5:77)7ed38dc09f787459979aebb39be/dist/codecov' failed with exit code 1

@another-rex
Copy link
Collaborator

We seem to be getting these errors every now and then, current workaround is just rerun the tests

Copy link
Collaborator

@another-rex another-rex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor variable naming nit, otherwise LGTM, thanks!

LoadPath string `toml:"LoadPath"`
GoVersionOverride string `toml:"GoVersionOverride"`
IgnoredVulns []IgnoreEntry `toml:"IgnoredVulns"`
PackageVersions []PackageVersionEntry `toml:"PackageVersions"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename this to PackageOverrides and PackageOverrideEntry, since version is optional, and we are specifying packages for the scanner to override the interpretation of

@josieang josieang enabled auto-merge (squash) June 6, 2024 03:55
@josieang josieang disabled auto-merge June 6, 2024 03:55
@josieang josieang force-pushed the package-config branch 2 times, most recently from 162fa59 to b7e6db1 Compare June 6, 2024 04:18
@josieang josieang merged commit 6d0e29e into google:main Jun 6, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants