Description

Attempt to fix #2930 and #2971.

This is a draft meant to receive feedback on each identified item as well as suggested changes, since this will be easier with actual commits visible.

This PR is an attempt to fix the 4 following issues:

1. A fact that has been loaded into an operation from a fact source will remain in the fact_store even after being deleted from the fact source. Consequences:

• When starting a new operation with the same fact source, the deleted fact will still be used.
• Unnecessary clutter that may become noticeable over time, potential for unintended data leak (agreed that Caldera shouldn't be used in a way that exposes sensitive data, but still).

2. A fact that has been loaded into an operation from a fact source will be duplicated in the fact_store when its value is updated in the fact source (very similar to the first item). Consequences:

• When starting a new operation with the same fact source, the fact will have both the old and the new value (or more if it is updated several times), and there is no way to stop that apart from creating a new fact source or fiddling with the fact_store manually.
• Unnecessary clutter that may become noticeable over time, potential for unintended data leak (agreed that Caldera shouldn't be used in a way that exposes sensitive data, but still).

3. A fact that has been loaded into an operation from a fact source will remain in the fact_store even when the fact source is deleted. Consequences:

• Unnecessary clutter that may become noticeable over time, potential for unintended data leak (agreed that Caldera shouldn't be used in a way that exposes sensitive data, but still).

4. A fact that has been generated by an operation (from parsers) will remain in the fact_store even after the operation is deleted, seemingly never to be used again. Consequences:

• Unnecessary clutter that may become noticeable over time, potential for unintended data leak (agreed that Caldera shouldn't be used in a way that exposes sensitive data, but still).

I will need some validation about :

• whether each item is or is not a bug and thus requires a fix or not.
• how we would like to fix each item, especially item 1 and 2 that are a bit more blurry.

Status (will be updated)

At this point, the commits are meant to fix item 3 and 4. Item 1 requires the modification of fact_api.py, delete_facts in a way that is less specific. Item 2 requires a bit more thinking and work. (I haven't had time to get to it yet, and I'd prefer to receive validation beforehand).

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Item 3: Deletion of data related to deleted fact sources:
  • Create a fact source with 1 fact/value, 1 rule, 1 relationship
  • Load it into an operation
  • Stop the server and check that the fact and the relationship have been added to the fact_store (rules are unaffected in my tests)
  • Restart the server and Delete the fact source
  • Stop the server and check that the fact and relationship have been deleted from the fact_store
• Item 4: Deletion of data related to deleted operations:
  • Create and launch an operation that generates facts and relationships
  • Stop the server and check that facts and relationships have been added to the fact_store
  • Restart the server and Delete the operation
  • Stop the server and check that the facts and relationships have been deleted from the fact_store

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works