Skip to content

Export cpg as json via Neo4j OGM #1215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 7, 2023
Merged

Export cpg as json via Neo4j OGM #1215

merged 3 commits into from
Nov 7, 2023

Conversation

peckto
Copy link
Collaborator

@peckto peckto commented Jun 15, 2023

This PR adds the functionality to export the cpg as json.
The serialization is done via the Neo4j OGM.
Basically the same as when importing to Neo4j Database, just without Database.
So the json export will suffer from known problems, like recursion and bigInt.

Json format might be useful for machine learning or other data science projects.

Proposed json format:

{
"nodes": [],
"edges": []
}

Example json file:
cpg.txt

  • compare json export with data from Neo4j Database import
  • add test case
  • support pre-save hooks

Closes #247 #776 #935

Sorry, something went wrong.

@peckto peckto force-pushed the json-export-ogm branch 4 times, most recently from dd078ff to 22d60fc Compare June 16, 2023 08:02
@peckto
Copy link
Collaborator Author

peckto commented Jun 20, 2023

The current implementation does not create any Neo4j Session object, which allows the json export to be used without a connection to the database server, which is of course desirable.
This has unfortunately the side effect, that pre-save hooks are not executed, because they are configured in the session.
I have currently three ideas how this could be fixed:
a) create some dummy session, without actual database connection
b) configure pre-save hooks without session
c) do not use the OGM pre-save hook functionality and instead do this by our self

Update: I now implemented option (c).

@KuechA
Copy link
Contributor

KuechA commented Jun 20, 2023

Related issues: #247 #776 #935 . We may want to enable customizing the output format somehow.

@peckto peckto force-pushed the json-export-ogm branch from 22d60fc to d0e9efe Compare June 21, 2023 08:56
@peckto
Copy link
Collaborator Author

peckto commented Jun 21, 2023

We may want to enable customizing the output format somehow

Good point! I created a separate function to export in the proposed json format (exportSimpleJson) .
In future, here one could step in to support other formats.

@peckto peckto force-pushed the json-export-ogm branch from d0e9efe to 7582a16 Compare June 21, 2023 10:11
@sonarqubecloud
Copy link

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

65.2% 65.2% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@konradweiss
Copy link
Collaborator

konradweiss commented Jun 22, 2023

  • Make a pass PrepareForNeo4jSerializationPass
  • Turn it on as a default pass for the Neo4j module.

Sorry, something went wrong.

@CLAassistant
Copy link

CLAassistant commented Oct 17, 2023

CLA assistant check
All committers have signed the CLA.

@peckto peckto force-pushed the json-export-ogm branch 3 times, most recently from ebb4d88 to 125ca8f Compare November 6, 2023 07:52
Copy link
Collaborator

@konradweiss konradweiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would change some names and add more explicit explanations in the form of function comments. Just to make it a bit more maintainable. I did also think whether or not it would be good to extract the export to json functionality into its own class to not overload the Application.kt but for now its fine so that is optional.

Verified

This commit was signed with the committer’s verified signature. The key has expired.
NeQuissimus Tim Steinbach

Verified

This commit was signed with the committer’s verified signature. The key has expired.
NeQuissimus Tim Steinbach

Verified

This commit was signed with the committer’s verified signature. The key has expired.
NeQuissimus Tim Steinbach
Copy link

sonarqubecloud bot commented Nov 7, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

85.3% 85.3% Coverage
0.0% 0.0% Duplication

@konradweiss konradweiss self-requested a review November 7, 2023 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Export cpg graph
4 participants