Importing snyk test scans produces error: An exception error occurred during the report import:'priority'

[AlBellom]

I am importing manually snyk scans created using snyk test. Occasionally I get the following error message:

An exception error occurred during the report import:'priority'

After I get the above error message, the remaining vulnerabilities are not imported.

I am using DefectDojo v. 2.29.4 hosted on an EC2 in AWS using the DefectDojo AMI.

I have some scan files that I could potentially share, however I would have to sanitize them first.

I took a look at the DD code, however since I am not familiar with the code and for amount of time I spent on it, I went as far as figuring out that the issue probably occurs in engagements/views.py, but I didn't determine where the "priority" string in the error message comes from.

Any help is greatly appreciated. Thank you.

[manuel-sommer]

Could you please provide a sample file to reproduce the issue?

[AlBellom]

The short answer is that I can't right now. I can't send the file as is, as I would have to sanitize the fields that contain some confidential information. Since I have over 100 vulns in that file, it would take me some time.

In the meantime it would be helpful if you can help me understand where the string "priority" that is part of the error message comes from. Is it related to the Jira "priority" field?

Also, I deleted all the imports for the Snyk organization I was having issues with, and re-imported the same file that was giving me the above error message and this time it worked fine. Not clear why. Thanks.

[manuel-sommer]

I don't know, do you have a full error, e.g. look at these examples:
#9838
#9837

In order to reproduce the issue, the right one vuln would be enough, you don't need to sanitize 100 vulns.

[AlBellom]

I have to figure out which vulnerability is causing the issue. I deleted the test for which the import was failing and re-import the same file with no errors this time. I can try to re-import it again and see if the issue persist or if it is gone.

Regardless, I have seen many instances of this issue in syslog. Here is an example:

2024/03/28 21:24:03 [28/Mar/2024 21:24:03] ERROR [dojo.api_v2.exception_handler:36] 'priority'
2024/03/28 21:24:03 Traceback (most recent call last):
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
2024/03/28 21:24:03 response = handler(request, *args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/rest_framework/mixins.py", line 19, in create
2024/03/28 21:24:03 self.perform_create(serializer)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/api_v2/views.py", line 3485, in perform_create
2024/03/28 21:24:03 serializer.save(push_to_jira=push_to_jira)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/api_v2/serializers.py", line 2502, in save
2024/03/28 21:24:03 ) = reimporter.reimport_scan(
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/importers/reimporter/reimporter.py", line 682, in reimport_scan
2024/03/28 21:24:03 ) = self.process_parsed_findings(
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/decorators.py", line 48, in wrapper
2024/03/28 21:24:03 return func(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/celery/local.py", line 182, in call
2024/03/28 21:24:03 return self._get_current_object()(*a, **kw)
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/celery/app/task.py", line 411, in call
2024/03/28 21:24:03 return self.run(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/importers/reimporter/reimporter.py", line 429, in process_parsed_findings
2024/03/28 21:24:03 finding.save(push_to_jira=push_to_jira)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/models.py", line 2943, in save
2024/03/28 21:24:03 finding_helper.post_process_finding_save(self, dedupe_option=dedupe_option, rules_option=rules_option, product_grading_option=product_grading_option,
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/decorators.py", line 75, in wrapper
2024/03/28 21:24:03 return func(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/decorators.py", line 48, in wrapper
2024/03/28 21:24:03 return func(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/celery/local.py", line 182, in call
2024/03/28 21:24:03 return self._get_current_object()(*a, **kw)
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/celery/app/task.py", line 411, in call
2024/03/28 21:24:03 return self.run(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/decorators.py", line 120, in wrapper
2024/03/28 21:24:03 return func(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/finding/helper.py", line 383, in post_process_finding_save
2024/03/28 21:24:03 jira_helper.push_to_jira(finding)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/jira_link/helper.py", line 602, in push_to_jira
2024/03/28 21:24:03 return update_jira_issue_for_finding(finding, *args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/decorators.py", line 75, in wrapper
2024/03/28 21:24:03 return func(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/decorators.py", line 48, in wrapper
2024/03/28 21:24:03 return func(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/celery/local.py", line 182, in call
2024/03/28 21:24:03 return self._get_current_object()(*a, **kw)
2024/03/28 21:24:03 File "/opt/dojo/lib/python3.10/site-packages/celery/app/task.py", line 411, in call
2024/03/28 21:24:03 return self.run(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/decorators.py", line 120, in wrapper
2024/03/28 21:24:03 return func(*args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/jira_link/helper.py", line 818, in update_jira_issue_for_finding
2024/03/28 21:24:03 return update_jira_issue(finding, *args, **kwargs)
2024/03/28 21:24:03 File "/opt/dojo/django-DefectDojo/./dojo/jira_link/helper.py", line 872, in update_jira_issue
2024/03/28 21:24:03 priority=fields['priority'],
2024/03/28 21:24:03 KeyError: 'priority'

Thanks again.

[manuel-sommer]

Hi @AlBellom , maybe this PR fixes your problem: #9571 ?
The release is 2.31.2 🌈
Please upgrade and try the latest version if this can be reproduced.

What is your opinion @Maffooch ? I did not look too deep into the code here.

[mtesauro]

@AlBellom
I agree that updating to the latest is the way forward.

The AMI uses godojo to do the initial install. You can use the instructions at https://github.com/DefectDojo/godojo/blob/master/docs-and-scripts/upgrading.md to upgrade an existing AMI install.

[AlBellom]

@mtesauro @manuel-sommer It appears the AMI doesn't have any dojo-stop script. Looking at systemctl, it seems like there is service called sysdojo.service that is able to start and stop the DefectDojo service, but it is not clear if also starts and stop all Celery services, the database, and whatever else. I didn't find any documentation around it, and I didn't dig any further into the sysdojo.service. Please advise. Thank you.

Update
I seems like the service /etc/systemd/system/sysdojo.service execute the binary /opt/dojo/systemd/sysdojo, which is not a script, so again, it is not quite clear if this command starts all the other DefectDojo component or just the app. It would be great if all this were documented. Thanks.

Also, what is first-install.service?

[mtesauro]

@AlBellom
You can get the dojo-stop & dojo-start scripts at https://github.com/DefectDojo/godojo/tree/master/docs-and-scripts

sysdojod is used to bring up DefectDojo as a Systemd service. You can use systemctl status sysdojod or pstree to see what processes are started by sysdojod. As I recall, it starts dojo and the celery processes (worker & beat)

first-install.service is exactly what it's named - its the service that brings up the website used to do the initial install of DefectDojo when you first launch the AMI. Once the first install is done, it's no longer needed and should just exit immediately if it is still being started on boot.

[AlBellom]

Thanks @mtesauro.

When I run /opt/dojo/bin/python3 ./manage.py migrate as part of the upgrade, I get the following message:

Running migrations:
No migrations to apply.
Your models in app(s): 'dojo' have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

It is not clear if this message is just a notice that there are no migrations or if there are further actions to be taken.

[mtesauro]

@AlBellom
You can always run showmigrations to make sure all the migrations have run. See https://docs.djangoproject.com/en/5.0/ref/django-admin/#django-admin-showmigrations

[AlBellom]

@mtesauro
There are few points in the upgrade instructions that are not correct:

1. dojo-stop works only if Dojo was started with dojo-start, as it uses screen. This is typically not the case for an AWS AMI, as Dojo is started at system boot.
2. I used sudo systemctl stop sysdojo to stop Dojo instead of dojo-stop.
3. The steps in (4), (5), and (6) should be executed as user dojosrv and therefore after running sudo su - dojosrv.
4. After I performed all the steps in the upgrading.md document, I restarted Dojo via sudo systemctl start sysdojo.
5. When I tried to access Dojo from the browser I got a 404...

Suggestions? Thanks.

Update
I was reading another post #9663, which talks about the issue of the AMI using Python 3.10 and Dojo using Python 3.11. The AMI where I have installed Dojo runs on Ubuntu 22.04.3 LTS and Python 3.10 and the installed Dojo 2.31.2. Could the issue I am experiencing related to the Pytjhon version?

[manuel-sommer]

Hi @AlBellom ,

coming back to the original problem. Do you still have the snyk issue and if yes, please provide a sample data. Then, I would retest it and make a PR to fix the bug. Otherwise, I would recommend to close this issue @mtesauro