Support custom fields in DefectDojo findings to fill specific custom fields in Jira

[FallenAtticus]

Description

In this PR I want to enhance the generic report importer of DefectDojo to extract some custom fields/columns from report files in order to send this custom fields to Jira to fill the mapped custom fields with the values from the report.

Let's assume you have a report with all of the required columns but also have some more columns you also want to have in your findings.

If you supply the following custom field mapping during the import / reimport:
{"customfield_12345": "CustomColumnA", "customfield_12346": "CustomColumnB"}

The importer would map the columns into a new json map with the custom field ids as key and the value of each report row as value. For example:
{"customfield_12345": "USD123.00", "customfield_12346": "96%"}

This json is stored in a new field in the finding table called "custom_fields".

Jira will merge the finding custom fields with existing product custom fields and send it to Jira.

Test results

I have added a bunch of unittests for this change.

Documentation

I am not sure if the documentation needs to be updated. Please let me know.

Checklist

This checklist is for your information.

• Make sure to rebase your PR against the very latest dev.
• Features/Changes should be submitted against the dev.
• Bugfixes should be submitted against the bugfix branch.
• Give a meaningful name to your PR, as it may end up being used in the release notes.
• Your code is flake8 compliant.
• Your code is python 3.11 compliant.
• If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
• Model changes must include the necessary migrations in the dojo/db_migrations folder.
• Add applicable tests to the unit tests.
• Add the proper label to categorize your PR.

[dryrunsecurity]

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security	Status	Findings
Configured Codepaths Analyzer	❌	42 findings
Sensitive Files Analyzer	✅	0 findings
AppSec Analyzer	✅	0 findings
Authn/Authz Analyzer	❕	1 finding
Secrets Analyzer	✅	0 findings

Note

🔴 Risk threshold exceeded. Adding a reviewer if one is configured in .dryrunsecurity.yaml.

notification list: @mtesauro @grendel513

Change Summary (click to expand)

The following is a summary of changes in this pull request made by me, your security buddy 🤖.
Note that this summary is auto-generated and not meant to be a definitive list of security issues
but rather a helpful summary from a security perspective.

Summary:

The changes in this pull request focus on enhancing the functionality and security of the DefectDojo application, particularly in the areas of Jira integration, custom field handling, and report parsing. The key changes include:

1. Improvements to the Jira integration, such as merging custom fields from the project/engagement and the finding when creating or updating a Jira issue, and handling the status of findings/groups in Jira.
2. Addition of custom fields mapping functionality, allowing users to map JIRA custom field IDs to report column names during import and reimport processes.
3. Enhancements to the CSV and JSON parsers, including the ability to handle custom fields and improve input validation.
4. Logging and error handling improvements to provide better visibility and troubleshooting capabilities.

From a security perspective, the changes appear to be focused on improving the overall robustness and security of the application. The handling of custom fields, the validation of user input, and the improvements to the Jira integration are all positive steps in ensuring the secure operation of the DefectDojo application.

However, it's important to ensure that all user-supplied data is properly validated and sanitized throughout the application to prevent potential security vulnerabilities, such as injection attacks or unintended data exposure. Additionally, the use of hardcoded values and the potential for sensitive information exposure should be carefully reviewed and addressed.

Files Changed:

1. dojo/engagement/views.py: Changes related to the processing of the Jira form, including the handling of the push_all_jira_issues and custom_fields_mapping values.
2. dojo/filters.py: Addition of a new custom_fields filter to the ApiFindingFilter and ReportFindingFilter classes, allowing users to search for findings based on their custom fields.
3. dojo/api_v2/serializers.py: Addition of a custom_fields_mapping field to the ImportScanSerializer and ReImportScanSerializer classes, enabling the mapping of JIRA custom field IDs to report column names.
4. dojo/db_migrations/0212_finding_custom_fields.py: Addition of a custom_fields field to the Finding model, allowing the storage of custom data related to findings.
5. dojo/models.py: Addition of a custom_fields field to the Finding model, similar to the changes in the migration file.
6. dojo/forms.py: Addition of a custom_fields_mapping field to the JIRAEngagementForm class, enabling the mapping of JIRA custom field IDs to report column names.
7. dojo/importers/base_importer.py: Changes to the parse_findings method to handle both static and dynamic test types, and updates to the parse_findings_static_test_type and parse_dynamic_test_type_tests methods to support additional keyword arguments.
8. dojo/jira_link/helper.py: Updates to the Jira integration functionality, including the handling of custom fields, finding status, and closed/resolved Jira issues.
9. dojo/templates/dojo/view_finding.html: Addition of a new column to display JIRA custom fields on the finding details page.
10. dojo/tools/generic/csv_parser.py: Addition of a custom_fields_mapping parameter to the _get_findings_csv function, allowing the extraction of custom fields from CSV reports.
11. dojo/templatetags/as_key_value_list.py: Introduction of a new Django template filter called as_key_value_list, which generates a string representation of a dictionary with each key-value pair on a new line.
12. dojo/tools/generic/parser.py: Updates to the GenericParser class to handle both CSV and JSON file formats, and the addition of a logger object.
13. unittests/scans/generic/generic_report5.json: Addition of two new findings to the existing JSON report.
14. dojo/tools/generic/json_parser.py: Enhancements to the `GenericJSONParser

Powered by DryRun Security

[Maffooch]

Hi @FallenAtticus thank you for producing this feature! I will discuss it with other moderators of the project to determine how we will proceed with it

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.