Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Add limited support and validation for version and revision #3657

Open
wants to merge 97 commits into
base: DAC-feature
Choose a base branch
from

Conversation

Mikaayenson
Copy link
Collaborator

@Mikaayenson Mikaayenson commented May 8, 2024

Issues

Resolves #3620

Summary

Adds limited support to validate use of the version or revision (added in 8.8) field within the TOML file. This approach leverages the toml path and simply checks if it's in a prebuilt rule directory, and raises an error if these fields are present in prebuilt rules. We could alternatively pass the path down and do validation earlier.

Bug Fix

  • Updates the default rules path in the detection rules _config to ../../ and adds a resolve() to complete the path. (this way the RulesConfig works for the custom dir and prebuilt rule directories)

Testing

  • The unit test should not be impacted at all (we test versioning throughout).
  • Run make test on a sample rule with the disallowed fields on prebuilt rules and custom rules.

Here are a series of commands used to test different aspects of the CLI:

- python -m detection_rules dev build-release --update-version-lock
- python -m detection_rules dev deprecate-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml
- python -m detection_rules dev update-lock-versions
- python -m detection_rules dev trim-version-lock 8.9.0
- python -m detection_rules kibana export-rules -d mikas_folder -s
- python -m detection_rules kibana import-rules -o -d mikas_folder/rules
- python -m detection_rules import-rules-to-repo /Users/stryker/Downloads/rules_export-8.ndjson -s mikas_folder/rules --required-only
- python -m detection_rules export-rules-from-repo
- python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml --api-format
- python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml --rule-format
Testing Prebuilt Rules

n-to-baseruledata) ✗ 
(detection-rules-build) ➜  detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗  cd /Users/stryker/workspace/Elastic/detection-rules ; /usr/bin/env /Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/bin/pyth
on /Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher 53138 -- -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/rules/windows/command_and_control_certreq_postdata.toml 
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Error loading rule in /Users/stryker/workspace/Elastic/detection-rules/rules/windows/command_and_control_certreq_postdata.toml
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 317, in run_module
    run_module_as_main(options.target, alter_argv=True)
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 238, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/main.py", line 230, in view_rule
    rule = RuleCollection().load_file(rule_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule_loader.py", line 436, in load_file
    return self.load_dict(obj, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule_loader.py", line 415, in load_dict
    rule = TOMLRule(path=path, contents=contents)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 6, in __init__
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule.py", line 1317, in __post_init__
    self.validate_version_revision()
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule.py", line 1360, in validate_version_revision
    raise ValueError(f"Prebuilt rule toml files in {dir_context} must not include `{error_fields}`. "
ValueError: Prebuilt rule toml files in /Users/stryker/workspace/Elastic/detection-rules/rules must not include `version and revision`. Rule Potential File Transfer via Certreq (79f0a1f7-ed6b-471c-8eb1-23abd6470b1c) includes version and revision.
(detection-rules-build) ➜  detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗ 

 cd /Users/stryker/workspace/Elastic/detection-rul%                                                                                                                                                                                                                               es ; /usr/bin/env /Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/bin/python /Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher 53148 -- -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/rules/windows/command_and_control_certreq_postdata.toml 
(detection-rules-build) ➜  detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗ 
(detection-rules-build) ➜  detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗  cd /Users/stryker/workspace/Elastic/detection-rules ; /usr/bin/env /Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/bin/pyth
on /Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher 53148 -- -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/rules/windows/command_and_control_certreq_postdata.toml 
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Error loading rule in /Users/stryker/workspace/Elastic/detection-rules/rules/windows/command_and_control_certreq_postdata.toml
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 317, in run_module
    run_module_as_main(options.target, alter_argv=True)
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 238, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/main.py", line 230, in view_rule
    rule = RuleCollection().load_file(rule_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule_loader.py", line 436, in load_file
    return self.load_dict(obj, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule_loader.py", line 415, in load_dict
    rule = TOMLRule(path=path, contents=contents)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 6, in __init__
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule.py", line 1317, in __post_init__
    self.validate_version_revision()
  File "/Users/stryker/workspace/Elastic/detection-rules/detection_rules/rule.py", line 1360, in validate_version_revision
    raise ValueError(f"Prebuilt rule toml files in {dir_context} must not include `{error_fields}`. "
ValueError: Prebuilt rule toml files in /Users/stryker/workspace/Elastic/detection-rules/rules must not include `revision`. Rule Potential File Transfer via Certreq (79f0a1f7-ed6b-471c-8eb1-23abd6470b1c) includes revision.
  • python -m detection_rules dev build-release --update-version-lock
    ...

A: a577e524-c2ee-47bd-9c5b-e917d01d3276, new version: 2
- min_stack_version added: 8.11.0
A: 947827c6-9ed6-4dec-903e-c856c86e72f3, new version: 3
- min_stack_version added: 8.3.0
A: f94e898e-94f1-4545-8923-03e4b2866211, new version: 1
- min_stack_version added: 8.8.0
A: 61336fe6-c043-4743-ab6e-41292f439603, new version: 1
- min_stack_version added: 8.3.0
A: ee53d67a-5f0c-423c-a53c-8084ae562b5c, new version: 2
- min_stack_version added: 8.3.0
A: 846fe13f-6772-4c83-bd39-9d16d4ad1a81, new version: 5
- min_stack_version added: 8.3.0
A: cf6995ec-32a9-4b2d-9340-f8e61acf3f4e, new version: 2
- min_stack_version added: 8.3.0
Updated /Users/stryker/workspace/Elastic/detection-rules/detection_rules/etc/version.lock.json file
Package saved to: /Users/stryker/workspace/Elastic/detection-rules/releases/8.15
loaded security_detection_engine manifests from the following package versions: ['8.14.1', '8.13.7', '8.13.6', '8.13.5', '8.13.4', '8.13.3', '8.13.2', '8.13.1', '8.12.12', '8.12.11', '8.12.10', '8.12.9', '8.12.8', '8.12.7', '8.12.6', '8.12.5', '8.12.4', '8.12.3', '8.12.2', '8.12.1', '8.11.16', '8.11.15', '8.11.14', '8.11.13', '8.11.12', '8.11.11', '8.11.10', '8.11.9', '8.11.8', '8.11.7', '8.11.6', '8.11.5', '8.11.4', '8.11.3', '8.11.2', '8.11.1', '8.10.18', '8.10.17', '8.10.16', '8.10.15', '8.10.14', '8.10.13', '8.10.12', '8.10.11', '8.10.10', '8.10.9', '8.10.8', '8.10.7', '8.10.6', '8.10.5', '8.10.4', '8.10.3', '8.10.2', '8.10.1', '8.9.15', '8.9.14', '8.9.13', '8.9.12', '8.9.11', '8.9.10', '8.9.9', '8.9.8', '8.9.7', '8.9.6', '8.9.5', '8.9.4', '8.9.3', '8.9.2', '8.9.1', '8.8.15', '8.8.14', '8.8.13', '8.8.12', '8.8.11', '8.8.10', '8.8.9', '8.8.8', '8.8.7', '8.8.6', '8.8.5', '8.8.4', '8.8.3', '8.8.2', '8.8.1', '8.7.13', '8.7.12', '8.7.11', '8.7.10', '8.7.9', '8.7.8', '8.7.7', '8.7.6', '8.7.5', '8.7.4', '8.7.3', '8.7.2', '8.7.1', '8.6.10', '8.6.9', '8.6.8', '8.6.7', '8.6.6', '8.6.5', '8.6.4', '8.6.3', '8.6.2', '8.6.1', '8.5.8', '8.5.7', '8.5.6', '8.5.5', '8.5.4', '8.5.3', '8.5.2', '8.5.1', '8.4.5', '8.4.4', '8.4.3', '8.4.2', '8.4.1', '8.3.4', '8.3.3', '8.3.2', '8.3.1', '8.2.1', '8.1.1', '1.0.2', '1.0.1']
[+] Adding historical rules from 8.14.1 package

  • sha256: b7c047a58f9a3acc01432dcff6bdffe188974b9face76d99364e2adf034136fb

  • 1108 rules included

  • python -m detection_rules dev deprecate-rule /Users/stryker/workspace/Elastic/detection-rules/rules/linux/command_and_control_cat_network_activity.toml
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Rule moved to /Users/stryker/workspace/Elastic/detection-rules/rules/linux/_deprecated/command_and_control_cat_network_activity.toml - remember to git add this file
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules dev update-lock-versions
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Are you sure you want to update hashes for 1107 rules without a version bump? [y/N]: y
No hashes updated
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules dev trim-version-lock 8.9.0
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules kibana export-rules -d mikas_folder -s
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

  • skipping Enumeration of Privileged Local Groups Membership - ValidationError

  • skipping test new terms rule - ValidationError

  • skipping test investigation_fields - ValidationError

  • skipping Suppression is not enabled, threshold fields not selected - KqlParseError

  • skipping Suppression is enabled, threshold fields selected - KqlParseError

  • skipping test rule - ValidationError

  • skipping test rule - ValidationError

  • skipping test2 - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping Potential Malicious File Downloaded from Google Drive - ValidationError

  • skipping Exchange Mailbox Export via PowerShell - ValidationError

  • skipping ertewr - ValidationError
    19 rules exported
    4 rules converted
    4 saved to mikas_folder
    15 errors saved to mikas_folder/_errors.txt
    (detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules kibana import-rules -o -d mikas_folder/rules
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 rule(s) successfully imported

  • 3bd3801f-33ca-42ec-8147-08bbc94c2f09
    (detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules import-rules-to-repo /Users/stryker/Downloads/rules_export-8.ndjson -s rules --required-only
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for rules/potential_malicious_file_downloaded_from_google_drive.toml

  • (detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗ python -m detection_rules export-rules-from-repo
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Exported 1113 rules into /Users/stryker/workspace/Elastic/detection-rules/exports/20240520T071312L.ndjson
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

Testing Custom Rules W/ bypass_version_lock: False

- python -m detection_rules dev build-release --update-version-lock

```bash
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building package 8.14
 - 0 rules excluded from package
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
Rule changes detected!
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
 - 0 changed rules
 - 1 new rules
 - 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
Rule changes detected!
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
 - 0 changed rules
 - 1 new rules
 - 0 newly deprecated rules
Detailed changes: 
  A: 30b0e855-6ebc-457e-85ed-c33e52d0c6fc, new version: 1
    - min_stack_version added: 8.8.0
Updated /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/etc/version.lock.json file
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
Package saved to: /Users/stryker/workspace/Elastic/detection-rules/releases/8.14
loaded security_detection_engine manifests from the following package versions: ['8.14.1', '8.13.7', '8.13.6', '8.13.5', '8.13.4', '8.13.3', '8.13.2', '8.13.1', '8.12.12', '8.12.11', '8.12.10', '8.12.9', '8.12.8', '8.12.7', '8.12.6', '8.12.5', '8.12.4', '8.12.3', '8.12.2', '8.12.1', '8.11.16', '8.11.15', '8.11.14', '8.11.13', '8.11.12', '8.11.11', '8.11.10', '8.11.9', '8.11.8', '8.11.7', '8.11.6', '8.11.5', '8.11.4', '8.11.3', '8.11.2', '8.11.1', '8.10.18', '8.10.17', '8.10.16', '8.10.15', '8.10.14', '8.10.13', '8.10.12', '8.10.11', '8.10.10', '8.10.9', '8.10.8', '8.10.7', '8.10.6', '8.10.5', '8.10.4', '8.10.3', '8.10.2', '8.10.1', '8.9.15', '8.9.14', '8.9.13', '8.9.12', '8.9.11', '8.9.10', '8.9.9', '8.9.8', '8.9.7', '8.9.6', '8.9.5', '8.9.4', '8.9.3', '8.9.2', '8.9.1', '8.8.15', '8.8.14', '8.8.13', '8.8.12', '8.8.11', '8.8.10', '8.8.9', '8.8.8', '8.8.7', '8.8.6', '8.8.5', '8.8.4', '8.8.3', '8.8.2', '8.8.1', '8.7.13', '8.7.12', '8.7.11', '8.7.10', '8.7.9', '8.7.8', '8.7.7', '8.7.6', '8.7.5', '8.7.4', '8.7.3', '8.7.2', '8.7.1', '8.6.10', '8.6.9', '8.6.8', '8.6.7', '8.6.6', '8.6.5', '8.6.4', '8.6.3', '8.6.2', '8.6.1', '8.5.8', '8.5.7', '8.5.6', '8.5.5', '8.5.4', '8.5.3', '8.5.2', '8.5.1', '8.4.5', '8.4.4', '8.4.3', '8.4.2', '8.4.1', '8.3.4', '8.3.3', '8.3.2', '8.3.1', '8.2.1', '8.1.1', '1.0.2', '1.0.1']
[+] Adding historical rules from 8.14.1 package
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This `version` will be ignored and defaulted to the version.lock.json file. Set `bypass_version_lock` to `True` in the rules config to use the TOML version.
- sha256: 05ba10377b4362b7472c0f5dee026bfc3d10e823cb6af814f902d4954ce2edde
- 1 rules included
  • python -m detection_rules dev deprecate-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Rule moved to /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/windows/_deprecated/potential_malicious_file_downloaded_from_google_drive.toml - remember to git add this file

  • python -m detection_rules dev update-lock-versions

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Are you sure you want to update hashes for 1 rules without a version bump? [y/N]: y
WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This version will be ignored and defaulted to the version.lock.json file. Set bypass_version_lock to True in the rules config to use the TOML version.
No hashes updated

  • python -m detection_rules dev trim-version-lock 8.9.0

  • python -m detection_rules kibana export-rules -d mikas_folder -s
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

  • skipping Enumeration of Privileged Local Groups Membership - ValidationError

  • skipping test new terms rule - ValidationError

  • skipping test investigation_fields - ValidationError

  • skipping Suppression is not enabled, threshold fields not selected - KqlParseError

  • skipping Suppression is enabled, threshold fields selected - KqlParseError

  • skipping test rule - ValidationError

  • skipping test rule - ValidationError

  • skipping test2 - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping Potential Malicious File Downloaded from Google Drive - ValidationError

  • skipping Potential Malicious File Downloaded from Google Drive - ValidationError

  • skipping Exchange Mailbox Export via PowerShell - ValidationError

  • skipping ertewr - ValidationError
    19 rules exported
    3 rules converted
    3 saved to mikas_folder
    16 errors saved to mikas_folder/_errors.txt

  • python -m detection_rules kibana import-rules -o -d mikas_folder/rules
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

WARNING: Rule Potential Malicious File Downloaded from Google Drive - 30b0e855-6ebc-457e-85ed-c33e52d0c6fc has a version set in the rule TOML. This version will be ignored and defaulted to the version.lock.json file. Set bypass_version_lock to True in the rules config to use the TOML version.
1 rule(s) successfully imported

  • 30b0e855-6ebc-457e-85ed-c33e52d0c6fc
    (detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules import-rules-to-repo /Users/stryker/Downloads/rules_export-8.ndjson -s mikas_folder/rules --required-only
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml

  • python -m detection_rules export-rules-from-repo
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Exported 1 rules into /Users/stryker/workspace/Elastic/detection-rules/exports/20240520T064059L.ndjson

  • python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml --api-format
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
"author": [
"Elastic"
],
"description": "Identifies potential malicious file download and execution from Google Drive. The rule checks for download activity from Google Drive URL, followed by the creation of files commonly leveraged by or for malware. This could indicate an attempt to run malicious scripts, executables or payloads.",
"index": [
"auditbeat-",
"logs-endpoint
"
],
"language": "eql",
"name": "Potential Malicious File Downloaded from Google Drive",
"query": "sequence by host.id, process.entity_id with maxspan=30s\n[any where\n\n /* Look for processes started or libraries loaded from untrusted or unsigned Windows, Linux or macOS binaries /\n (event.action in ("exec", "fork", "start", "load")) or\n\n / Look for Google Drive download URL with AV flag skipping /\n (process.args : "drive.google.com" and process.args : "export=download" and process.args : "confirm=no_antivirus")\n]\n\n[network where\n / Look for DNS requests for Google Drive /\n (dns.question.name : "drive.google.com" and dns.question.type : "A") or\n\n / Look for connection attempts to address that resolves to Google /\n (destination.as.organization.name : "GOOGLE" and event.action == "connection_attempted")\n\n / NOTE: Add LoLBins if tuning is required\n process.name : (\n "cmd.exe", "bitsadmin.exe", "certutil.exe", "esentutl.exe", "wmic.exe", "PowerShell.exe",\n "homedrive.exe","regsvr32.exe", "mshta.exe", "rundll32.exe", "cscript.exe", "wscript.exe",\n "curl", "wget", "scp", "ftp", "python", "perl", "ruby"))] /\n]\n\n/ Identify the creation of files following Google Drive connection with extensions commonly used for executables or libraries */\n[file where event.action == "creation" and file.extension : (\n "exe", "dll", "scr", "jar", "pif", "app", "dmg", "pkg", "elf", "so", "bin", "deb", "rpm","sh","hta","lnk"\n )\n]\n",
"required_fields": [
{
"ecs": true,
"name": "destination.as.organization.name",
"type": "keyword"
},
{
"ecs": true,
"name": "dns.question.name",
"type": "keyword"
},
{
"ecs": true,
"name": "dns.question.type",
"type": "keyword"
},
{
"ecs": true,
"name": "event.action",
"type": "keyword"
},
{
"ecs": true,
"name": "file.extension",
"type": "keyword"
},
{
"ecs": true,
"name": "host.id",
"type": "keyword"
},
{
"ecs": true,
"name": "process.args",
"type": "keyword"
},
{
"ecs": true,
"name": "process.entity_id",
"type": "keyword"
}
],
"risk_score": 73,
"rule_id": "3bd3801f-33ca-42ec-8147-08bbc94c2f09",
"severity": "high",
"type": "eql",
"version": 1
}

  • python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml --rule-format
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[metadata]
creation_date = "2024/05/20"
maturity = "development"
updated_date = "2024/05/20"

[rule]
author = ["Elastic"]
description = """
Identifies potential malicious file download and execution from Google Drive. The rule checks for download activity from
Google Drive URL, followed by the creation of files commonly leveraged by or for malware. This could indicate an attempt
to run malicious scripts, executables or payloads.
"""
index = ["auditbeat-", "logs-endpoint"]
language = "eql"
name = "Potential Malicious File Downloaded from Google Drive"
risk_score = 73
rule_id = "3bd3801f-33ca-42ec-8147-08bbc94c2f09"
severity = "high"
type = "eql"

query = '''
sequence by host.id, process.entity_id with maxspan=30s
[any where

/* Look for processes started or libraries loaded from untrusted or unsigned Windows, Linux or macOS binaries */
(event.action in ("exec", "fork", "start", "load")) or

/* Look for Google Drive download URL with AV flag skipping */
(process.args : "*drive.google.com*" and process.args : "*export=download*" and process.args : "*confirm=no_antivirus*")

]

[network where
/* Look for DNS requests for Google Drive */
(dns.question.name : "drive.google.com" and dns.question.type : "A") or

/* Look for connection attempts to address that resolves to Google */
(destination.as.organization.name : "GOOGLE" and event.action == "connection_attempted")

/* NOTE: Add LoLBins if tuning is required
process.name : (
    "cmd.exe", "bitsadmin.exe", "certutil.exe", "esentutl.exe", "wmic.exe", "PowerShell.exe",
    "homedrive.exe","regsvr32.exe", "mshta.exe", "rundll32.exe", "cscript.exe", "wscript.exe",
    "curl", "wget", "scp", "ftp", "python", "perl", "ruby"))] */

]

/* Identify the creation of files following Google Drive connection with extensions commonly used for executables or libraries */
[file where event.action == "creation" and file.extension : (
"exe", "dll", "scr", "jar", "pif", "app", "dmg", "pkg", "elf", "so", "bin", "deb", "rpm","sh","hta","lnk"
)
]
'''

Testing Custom Rules W/ bypass_version_lock: True

- python -m detection_rules dev build-release --update-version-lock Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building package 8.14
Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.

  • 1 rules excluded from package
    WARNING: Cannot bypass version lock. Version lock is enabled. Set bypass_version_lock to false in the rules config to use the version lock.
    WARNING: Version Lock usage is bypassed. Set bypass_version_lock: false in the rules config to enable.
    WARNING: You cannot update the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.

  • python -m detection_rules dev deprecate-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Cannot deprecate a rule when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules dev update-lock-versions
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
Are you sure you want to update hashes for 0 rules without a version bump? [y/N]: y
WARNING: You cannot run this command when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to False in the rules config to use the version lock.
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules dev trim-version-lock 8.9.0
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Cannot trim the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules kibana export-rules -d mikas_folder -s
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

  • skipping Enumeration of Privileged Local Groups Membership - ValidationError

  • skipping test new terms rule - ValidationError

  • skipping test investigation_fields - ValidationError

  • skipping Suppression is not enabled, threshold fields not selected - KqlParseError

  • skipping Suppression is enabled, threshold fields selected - KqlParseError

  • skipping test rule - ValidationError

  • skipping test rule - ValidationError

  • skipping test2 - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError

  • skipping Potential Malicious File Downloaded from Google Drive - ValidationError

  • skipping Exchange Mailbox Export via PowerShell - ValidationError

  • skipping ertewr - ValidationError
    19 rules exported
    4 rules converted
    4 saved to mikas_folder
    15 errors saved to mikas_folder/_errors.txt
    (detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules kibana import-rules -o -d mikas_folder/rules
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
1 rule(s) successfully imported

  • 3bd3801f-33ca-42ec-8147-08bbc94c2f09
    (detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules import-rules-to-repo /Users/stryker/Downloads/rules_export-8.ndjson -s mikas_folder/rules --required-only
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules export-rules-from-repo
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
Exported 1 rules into /Users/stryker/workspace/Elastic/detection-rules/exports/20240520T070113L.ndjson

  • python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml --api-format
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
{
"author": [
"Elastic"
],
"description": "Identifies potential malicious file download and execution from Google Drive. The rule checks for download activity from Google Drive URL, followed by the creation of files commonly leveraged by or for malware. This could indicate an attempt to run malicious scripts, executables or payloads.",
"index": [
"auditbeat-",
"logs-endpoint
"
],
"language": "eql",
"name": "Potential Malicious File Downloaded from Google Drive",
"query": "sequence by host.id, process.entity_id with maxspan=30s\n[any where\n\n /* Look for processes started or libraries loaded from untrusted or unsigned Windows, Linux or macOS binaries /\n (event.action in ("exec", "fork", "start", "load")) or\n\n / Look for Google Drive download URL with AV flag skipping /\n (process.args : "drive.google.com" and process.args : "export=download" and process.args : "confirm=no_antivirus")\n]\n\n[network where\n / Look for DNS requests for Google Drive /\n (dns.question.name : "drive.google.com" and dns.question.type : "A") or\n\n / Look for connection attempts to address that resolves to Google /\n (destination.as.organization.name : "GOOGLE" and event.action == "connection_attempted")\n\n / NOTE: Add LoLBins if tuning is required\n process.name : (\n "cmd.exe", "bitsadmin.exe", "certutil.exe", "esentutl.exe", "wmic.exe", "PowerShell.exe",\n "homedrive.exe","regsvr32.exe", "mshta.exe", "rundll32.exe", "cscript.exe", "wscript.exe",\n "curl", "wget", "scp", "ftp", "python", "perl", "ruby"))] /\n]\n\n/ Identify the creation of files following Google Drive connection with extensions commonly used for executables or libraries */\n[file where event.action == "creation" and file.extension : (\n "exe", "dll", "scr", "jar", "pif", "app", "dmg", "pkg", "elf", "so", "bin", "deb", "rpm","sh","hta","lnk"\n )\n]\n",
"required_fields": [
{
"ecs": true,
"name": "destination.as.organization.name",
"type": "keyword"
},
{
"ecs": true,
"name": "dns.question.name",
"type": "keyword"
},
{
"ecs": true,
"name": "dns.question.type",
"type": "keyword"
},
{
"ecs": true,
"name": "event.action",
"type": "keyword"
},
{
"ecs": true,
"name": "file.extension",
"type": "keyword"
},
{
"ecs": true,
"name": "host.id",
"type": "keyword"
},
{
"ecs": true,
"name": "process.args",
"type": "keyword"
},
{
"ecs": true,
"name": "process.entity_id",
"type": "keyword"
}
],
"risk_score": 73,
"rule_id": "3bd3801f-33ca-42ec-8147-08bbc94c2f09",
"severity": "high",
"type": "eql",
"version": 1
}
(detection-rules-build) ➜ detection-rules git:(3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata) ✗

  • python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/mikas_folder/rules/potential_malicious_file_downloaded_from_google_drive.toml --rule-format
    Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Cannot set the version lock when the versioning strategy is configured to bypass the version lock. Set bypass_version_lock to false in the rules config to use the version lock.
[metadata]
creation_date = "2024/05/20"
maturity = "development"
updated_date = "2024/05/20"

[rule]
author = ["Elastic"]
description = """
Identifies potential malicious file download and execution from Google Drive. The rule checks for download activity from
Google Drive URL, followed by the creation of files commonly leveraged by or for malware. This could indicate an attempt
to run malicious scripts, executables or payloads.
"""
index = ["auditbeat-", "logs-endpoint"]
language = "eql"
name = "Potential Malicious File Downloaded from Google Drive"
risk_score = 73
rule_id = "3bd3801f-33ca-42ec-8147-08bbc94c2f09"
severity = "high"
type = "eql"

query = '''
sequence by host.id, process.entity_id with maxspan=30s
[any where

/* Look for processes started or libraries loaded from untrusted or unsigned Windows, Linux or macOS binaries */
(event.action in ("exec", "fork", "start", "load")) or

/* Look for Google Drive download URL with AV flag skipping */
(process.args : "*drive.google.com*" and process.args : "*export=download*" and process.args : "*confirm=no_antivirus*")

]

[network where
/* Look for DNS requests for Google Drive */
(dns.question.name : "drive.google.com" and dns.question.type : "A") or

/* Look for connection attempts to address that resolves to Google */
(destination.as.organization.name : "GOOGLE" and event.action == "connection_attempted")

/* NOTE: Add LoLBins if tuning is required
process.name : (
    "cmd.exe", "bitsadmin.exe", "certutil.exe", "esentutl.exe", "wmic.exe", "PowerShell.exe",
    "homedrive.exe","regsvr32.exe", "mshta.exe", "rundll32.exe", "cscript.exe", "wscript.exe",
    "curl", "wget", "scp", "ftp", "python", "perl", "ruby"))] */

]

/* Identify the creation of files following Google Drive connection with extensions commonly used for executables or libraries */
[file where event.action == "creation" and file.extension : (
"exe", "dll", "scr", "jar", "pif", "app", "dmg", "pkg", "elf", "so", "bin", "deb", "rpm","sh","hta","lnk"
)
]
'''

Version Strategy Warning

  • General (bypass_version_lock = False)
    • Default
    • Versions from Kibana or the TOML file are ignored
    • Version lock file usage is permitted
  • General (bypass_version_lock = True)
    • Must be explicitly set in the config
    • Versions from Kibana or the TOML file are used
    • Version lock file usage is not permitted
  • Tactical Warning Messages
    • Rule import to TOML file will skip version and revision fields when supplied (rule_prompt & import_rules_into_repo) if bypass_version_lock = False. No warning message is issued.
    • Rule version lock will not be updated or used if bypass_version_lock = True when building a release package (build_release). A warning message is issued.
    • If versions are in the TOML file, and bypass_version_lock = False, the versions in the TOML file will not be used (autobumped_version). A warning message is issued.
    • If bypass_version_lock = False, when autobumping the version, it will check the version lock file and increment if is_dirty (autobumped_version), otherwise just use the version supplied. No warning message is issued.
    • If bypass_version_lock = True, the updating the version lock file will disabled (update_lock_versions). A warning message is issued.
    • If bypass_version_lock = True, loading the version lock file is disabled and skipped. (from_dict, load_from_file, manage_versions, test_version_lock_has_nested_previous). A warning message is issued.

@Mikaayenson Mikaayenson added enhancement New feature or request python Internal python for the repository detections-as-code labels May 8, 2024
@Mikaayenson Mikaayenson requested a review from traut May 8, 2024 22:54
…-to-baseruledata' of github.com:elastic/detection-rules into 3620-frdac-add-limited-support-for-version-and-revision-to-baseruledata
@eric-forte-elastic
Copy link
Collaborator

eric-forte-elastic commented May 23, 2024

Testing again with recent changes:

Bypass Version Lock False 🟢

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev build-release --update-version-lock
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/devtools.py", line 98, in build_release
    registry_data = config['registry_data']
                    ~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'registry_data'

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev deprecate-rule custom_rules/rules/command_and_control_test_potential_protocol_tunneling_via_chisel_client.toml
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Rule has not been version locked and so does not need to be deprecated. Delete the file or update the maturity to `development` instead.

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev update-lock-versions
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Are you sure you want to update hashes for 4 rules without a version bump? [y/N]: Y
Rule changes detected!
 - 0 changed rules
 - 4 new rules
 - 0 newly deprecated rules
Detailed changes:

Updated /home/forteea1/Code/clean_mains/detection-rules/custom_rules/etc/version.lock.json file
No hashes updated

❯ python -m detection_rules dev trim-version-lock 8.15.0
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Loading rules ...
No changes


…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev trim-version-lock 8.9.0
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Loading rules ...
No changes


…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules kibana export-rules -d mikas_folder -s
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

- skipping DAC Demo Dev Rule 1 - also updated - ValidationError
- skipping DAC Demo Dev Rule 13 - ValidationError
- skipping DAC Demo Dev Rule 1 - ValidationError
- skipping TEST Potential Protocol Tunneling via Chisel Client - ValidationError
4 rules exported
0 rules converted
0 saved to mikas_folder
4 errors saved to mikas_folder/_errors.txt

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 2s
❯ python -m detection_rules kibana export-rules -d custom_rules -s
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

- skipping DAC Demo Dev Rule 1 - also updated - ValidationError
- skipping DAC Demo Dev Rule 13 - ValidationError
- skipping DAC Demo Dev Rule 1 - ValidationError
- skipping TEST Potential Protocol Tunneling via Chisel Client - ValidationError
4 rules exported
0 rules converted
0 saved to custom_rules
4 errors saved to custom_rules/_errors.txt

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 2s
❯ cat custom_rules/_errors.txt
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: custom_rules/_errors.txt
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ - DAC Demo Dev Rule 1 - also updated - {'rule': [ValidationError({'_schema': ['Configuration error: Rule DAC Demo Dev
       │ Rule 1 - also updated - 09ea590e-69d1-44de-a386-3d84d85cc2f2 should not contain rules with `version and revision` set.
']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({
       │ 'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type':
       │  ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing
       │  data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': [
       │ 'Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationErro
       │ r({'type': ['Must be equal to query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Mis
       │ sing data for required field.']})]}
   2   │ - DAC Demo Dev Rule 13 - {'rule': [ValidationError({'_schema': ['Configuration error: Rule DAC Demo Dev Rule 13 - 222e
       │ 1b03-fdc9-42a5-911e-2e3e0533549a should not contain rules with `version and revision` set.']}), ValidationError({'type
': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to t
       │ hreshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_m
       │ atch.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}
       │ ), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required f
       │ ield.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal t
       │ o query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field
       │ .']})]}
   3   │ - DAC Demo Dev Rule 1 - {'rule': [ValidationError({'_schema': ['Configuration error: Rule DAC Demo Dev Rule 1 - 794d2f
       │ c0-ecd0-4963-99da-fd587666b80d should not contain rules with `version and revision` set.']}), ValidationError({'type':
       │  ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to thr
       │ eshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_mat
       │ ch.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}),
       │  ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required fie
       │ ld.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to
       │ query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.'
       │ ]})]}
   4   │ - TEST Potential Protocol Tunneling via Chisel Client - {'rule': [ValidationError({'_schema': ['Configuration error: R
       │ ule TEST Potential Protocol Tunneling via Chisel Client - e8e3af2a-11b8-4ab7-9ca1-c6db621ea89d should not contain rule
       │ s with `version and revision` set.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equ
       │ al to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required fi
       │ eld.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required f
       │ ield.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_l
       │ earning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for r
       │ equired field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'type': ['Must be equal t
       │ o new_terms.'], 'new_terms': ['Missing data for required field.']})]}
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules kibana export-rules -d custom_rules/rules -s -sv
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

4 rules exported
4 rules converted
4 saved to custom_rules/rules

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 3s
❯ python -m detection_rules kibana import-rules -o -d custom_rules/rules/
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

4 rule(s) successfully imported
 - 794d2fc0-ecd0-4963-99da-fd587666b80d
 - 222e1b03-fdc9-42a5-911e-2e3e0533549a
 - e8e3af2a-11b8-4ab7-9ca1-c6db621ea89d
 - 09ea590e-69d1-44de-a386-3d84d85cc2f2

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 5s
❯ python -m detection_rules import-rules-to-repo /home/forteea1/Code/dac_demo/detection-rules-dac-demo/exports/20240518T115752L.ndjson -s custom_rules/rules --required-only
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for custom_rules/rules/dac_demo_dev_rule_1_updated.toml
[+] Building rule for custom_rules/rules/dac_demo_dev_rule_2_feedback.toml

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules export-rules-from-repo
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Exported 6 rules into /home/forteea1/Code/clean_mains/detection-rules/exports/20240523T161054L.ndjson

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules view-rule rules/cross-platform/threat_intel_indicator_match_address.toml
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "This rule is triggered when an IP address indicator from the Threat Intel Filebeat module or integrations has a match against a network event.",
  "from": "now-65m",
  "index": [
    "auditbeat-*",
    "endgame-*",
    "filebeat-*",
    "logs-*",
    "packetbeat-*",
    "winlogbeat-*"
  ],
  "interval": "1h",
  "language": "kuery",
  "license": "Elastic License v2",
  "name": "Threat Intel IP Address Indicator Match",
  "note": "## Triage and Analysis\n\n### Investigating Threat Intel IP Address Indicator Match\n\nThreat Intel indicator match rules allow matching from a local observation, such as an endpoint event that records a file hash with an entry of a file hash stored within the Threat Intel integrations index.\n\nMatches are based on threat intelligence data that's been ingested during the last 30 days. Some integrations don't place expiration dates on their threat indicators, so we strongly recommend validating ingested threat indicators and reviewing match results. When reviewing match results, check associated activity to determine whether the event requires additional investigation.\n\nThis rule is triggered when an IP address indicator from the Threat Intel Filebeat module or a threat intelligence integration matches against a network event.\n\n> **Note**:\n> This investigation guide uses the [Osquery Markdown Plugin](https://www.elastic.co/guide/en/security/master/invest-guide-run-osquery.html) introduced in Elastic Stack version 8.5.0. Older Elastic Stack versions will display unrendered Markdown in this guide.\n\n#### Possible investigation steps\n\n- Gain context about the field that matched the local observation so you can understand the nature of the connection. This information can be found in the `threat.indicator.matched.field` field.\n- Investigate the IP address, which can be found in the `threat.indicator.matched.atomic` field:\n  - Check the reputation of the IP address in resources like VirusTotal, Hybrid-Analysis, CISCO Talos, Any.run, etc.\n  - Execute a reverse DNS lookup to retrieve hostnames associated with the given IP address.\n- Assess whether this behavior is prevalent in the environment by looking for similar occurrences across hosts.\n- Identify the process responsible for the connection, and investigate the process execution chain (parent process tree) for unknown processes. Examine their executable files for prevalence, whether they are located in expected locations, and if they are signed with valid digital signatures.\n- Retrieve the involved process executable and examine the host for derived artifacts that indicate suspicious activities:\n  - Analyze the process executable using a private sandboxed analysis system.\n  - Observe and collect information about the following activities in both the sandbox and the alert subject host:\n    - Attempts to contact external domains and addresses.\n      - Use the Elastic Defend network events to determine domains and addresses contacted by the subject process by filtering by the process' `process.entity_id`.\n      - Examine the DNS cache for suspicious or anomalous entries.\n        - !{osquery{\"label\":\"Osquery - Retrieve DNS Cache\",\"query\":\"SELECT * FROM dns_cache\"}}\n    - Use the Elastic Defend registry events to examine registry keys accessed, modified, or created by the related processes in the process tree.\n    - Examine the host services for suspicious or anomalous entries.\n      - !{osquery{\"label\":\"Osquery - Retrieve All Services\",\"query\":\"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services\"}}\n      - !{osquery{\"label\":\"Osquery - Retrieve Services Running on User Accounts\",\"query\":\"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services WHERE\\nNOT (user_account LIKE '%LocalSystem' OR user_account LIKE '%LocalService' OR user_account LIKE '%NetworkService' OR\\nuser_account == null)\\n\"}}\n      - !{osquery{\"label\":\"Osquery - Retrieve Service Unsigned Executables with Virustotal Link\",\"query\":\"SELECT concat('https://www.virustotal.com/gui/file/', sha1) AS VtLink, name, description, start_type, status, pid,\\nservices.path FROM services JOIN authenticode ON services.path = authenticode.path OR services.module_path =\\nauthenticode.path JOIN hash ON services.path = hash.path WHERE authenticode.result != 'trusted'\\n\"}}\n- Using the data collected through the analysis, scope users targeted and other machines infected in the environment.\n\n### False Positive Analysis\n\n- When a match is found, it's important to consider the indicator's initial release date. Threat intelligence is useful for augmenting existing security processes but can quickly become outdated. In other words, some threat intelligence only represents a specific set of activity observed at a specific time. For example, an IP address may have hosted malware observed in a Dridex campaign months ago, but it's possible that IP has been remediated and no longer represents any threat.\n- False positives might occur after large and publicly written campaigns if curious employees interact with attacker infrastructure.\n- Some feeds may include internal or known benign addresses by mistake (e.g., 8.8.8.8, google.com, 127.0.0.1, etc.). Make sure you understand how blocking a specific domain or address might impact the organization or normal system functioning.\n\n### Response and Remediation\n\n- Initiate the incident response process based on the outcome of the triage.\n- Isolate the involved host to prevent further post-compromise behavior.\n- If the triage identified malware, search the environment for additional compromised hosts.\n  - Implement temporary network rules, procedures, and segmentation to contain the malware.\n  - Stop suspicious processes.\n  - Immediately block the identified indicators of compromise (IoCs).\n  - Inspect the affected systems for additional malware backdoors like reverse shells, reverse proxies, or droppers that attackers could use to reinfect the system.\n- Remove and block malicious artifacts identified during triage.\n- Run a full antimalware scan. This may reveal additional artifacts left in the system, persistence mechanisms, and malware components.\n- Determine the initial vector abused by the attacker and take action to prevent reinfection through the same vector.\n- Using the incident response data, update logging and audit policies to improve the mean time to detect (MTTD) and the mean time to respond (MTTR).\n",
  "query": "source.ip:* or destination.ip:*\n",
  "references": [
    "https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-threatintel.html",
    "https://www.elastic.co/guide/en/security/master/es-threat-intel-integrations.html",
    "https://www.elastic.co/security/tip"
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "destination.ip",
      "type": "ip"
    },
    {
      "ecs": true,
      "name": "source.ip",
      "type": "ip"
    }
  ],
  "risk_score": 99,
  "rule_id": "0c41e478-5263-4c69-8f9e-7dfd2c22da64",
  "setup": "## Setup\n\nThis rule needs threat intelligence indicators to work.\nThreat intelligence indicators can be collected using an [Elastic Agent integration](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#agent-ti-integration),\nthe [Threat Intel module](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#ti-mod-integration),\nor a [custom integration](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html#custom-ti-integration).\n\nMore information can be found [here](https://www.elastic.co/guide/en/security/current/es-threat-intel-integrations.html).\n",
  "severity": "critical",
  "tags": [
    "OS: Windows",
    "Data Source: Elastic Endgame",
    "Rule Type: Indicator Match"
  ],
  "threat_filters": [
    {
      "$state": {
        "store": "appState"
      },
      "meta": {
        "disabled": false,
        "key": "event.category",
        "negate": false,
        "params": {
          "query": "threat"
        },
        "type": "phrase"
      },
      "query": {
        "match_phrase": {
          "event.category": "threat"
        }
      }
    },
    {
      "$state": {
        "store": "appState"
      },
      "meta": {
        "disabled": false,
        "key": "event.kind",
        "negate": false,
        "params": {
          "query": "enrichment"
        },
        "type": "phrase"
      },
      "query": {
        "match_phrase": {
          "event.kind": "enrichment"
        }
      }
    },
    {
      "$state": {
        "store": "appState"
      },
      "meta": {
        "disabled": false,
        "key": "event.type",
        "negate": false,
        "params": {
          "query": "indicator"
        },
        "type": "phrase"
      },
      "query": {
        "match_phrase": {
          "event.type": "indicator"
        }
      }
    }
  ],
  "threat_index": [
    "filebeat-*",
    "logs-ti_*"
  ],
  "threat_indicator_path": "threat.indicator",
  "threat_language": "kuery",
  "threat_mapping": [
    {
      "entries": [
        {
          "field": "source.ip",
          "type": "mapping",
          "value": "threat.indicator.ip"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "destination.ip",
          "type": "mapping",
          "value": "threat.indicator.ip"
        }
      ]
    }
  ],
  "threat_query": "@timestamp >= \"now-30d/d\" and event.module:(threatintel or ti_*) and threat.indicator.ip:* and not labels.is_ioc_transform_source:\"true\"",
  "timeline_id": "495ad7a7-316e-4544-8a0f-9c098daee76e",
  "timeline_title": "Generic Threat Match Timeline",
  "timestamp_override": "event.ingested",
  "type": "threat_match",
  "version": 1
}

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules view-rule custom_rules/rules/command_and_control_dac_demo_dev_rule_1.toml --api-format
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "actions": [],
  "author": [
    "DAC User"
  ],
  "description": "Test Rule",
  "enabled": true,
  "exceptions_list": [],
  "false_positives": [],
  "from": "now-9m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "interval": "5m",
  "language": "eql",
  "license": "Elastic License v2",
  "max_signals": 100,
  "name": "DAC Demo Dev Rule 1",
  "note": "Test Note",
  "query": "process where host.os.type == \"linux\" and process.name == \"updated\"\n",
  "references": [
    "https://dac-reference.readthedocs.io/en/latest/"
  ],
  "related_integrations": [
    {
      "package": "endpoint",
      "version": "^8.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "host.os.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    }
  ],
  "risk_score": 47,
  "risk_score_mapping": [],
  "rule_id": "794d2fc0-ecd0-4963-99da-fd587666b80d",
  "setup": "Test Setup",
  "severity": "medium",
  "severity_mapping": [],
  "tags": [
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Command and Control",
    "Data Source: Elastic Defend"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0011",
        "name": "Command and Control",
        "reference": "https://attack.mitre.org/tactics/TA0011/"
      },
      "technique": [
        {
          "id": "T1572",
          "name": "Protocol Tunneling",
          "reference": "https://attack.mitre.org/techniques/T1572/"
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "to": "now",
  "type": "eql",
  "version": 1
}

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules view-rule custom_rules/rules/command_and_control_dac_demo_dev_rule_1.toml --rule-format
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[metadata]
creation_date = "2024/05/23"
maturity = "production"
updated_date = "2024/05/23"

[rule]
actions = []
author = ["DAC User"]
description = "Test Rule"
enabled = true
exceptions_list = []
false_positives = []
from = "now-9m"
index = ["logs-endpoint.events.*"]
interval = "5m"
language = "eql"
license = "Elastic License v2"
max_signals = 100
name = "DAC Demo Dev Rule 1"
note = "Test Note"
references = ["https://dac-reference.readthedocs.io/en/latest/"]
risk_score = 47
risk_score_mapping = []
rule_id = "794d2fc0-ecd0-4963-99da-fd587666b80d"
setup = "Test Setup"
severity = "medium"
severity_mapping = []
tags = [
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Command and Control",
    "Data Source: Elastic Defend",
]
timestamp_override = "event.ingested"
to = "now"
type = "eql"

query = '''
process where host.os.type == "linux" and process.name == "updated"
'''


[[rule.related_integrations]]
package = "endpoint"
version = "^8.2.0"

[[rule.required_fields]]
ecs = true
name = "host.os.type"
type = "keyword"

[[rule.required_fields]]
ecs = true
name = "process.name"
type = "keyword"

[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1572"
name = "Protocol Tunneling"
reference = "https://attack.mitre.org/techniques/T1572/"


[rule.threat.tactic]
id = "TA0011"
name = "Command and Control"
reference = "https://attack.mitre.org/tactics/TA0011/"



…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules view-rule custom_rules/rules/command_and_control_dac_demo_dev_rule_1.toml --rule-format^C

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev build-release --update-version-lock
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/devtools.py", line 101, in build_release
    assert 'registry_data' in config, err_msg
           ^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: No `registry_data` in package config. Please see the /home/forteea1/Code/clean_mains/detection-rules/detection_rules/etc/package.yaml file for an example on how to supply this field in /home/forteea1/Code/clean_mains/detection-rules/custom_rules/etc/packages.yaml.


Version Lock True 🟢

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev build-release --update-version-lock
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

WARNING: You cannot run this command when the versioning strategy is configured to bypass the version lock. Set `bypass_version_lock` to `False` in the rules config to use the version lock.

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev deprecate-rule custom_rules/rules/command_and_control_test_potential_protocol_tunneling_via_chisel_client.toml
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Rule moved to /home/forteea1/Code/clean_mains/detection-rules/custom_rules/rules/_deprecated/command_and_control_test_potential_protocol_tunneling_via_chisel_client.toml - remember to git add this file

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules dev update-lock-versions
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Are you sure you want to update hashes for 3 rules without a version bump? [y/N]: Y
WARNING: You cannot run this command when the versioning strategy is configured to bypass the version lock. Set `bypass_version_lock` to `False` in the rules config to use the version lock.

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 8s
❯ python -m detection_rules dev trim-version-lock 8.9.0
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

WARNING: Cannot trim the version lock when the versioning strategy is configured to bypass the version lock. Set `bypass_version_lock` to `false` in the rules config to use the version lock.

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules kibana export-rules -d custom_rules/rules -s -sv
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

4 rules exported
4 rules converted
4 saved to custom_rules/rules

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 4s
❯ python -m detection_rules kibana export-rules -d custom_rules/rules -s
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

4 rules exported
4 rules converted
4 saved to custom_rules/rules

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 2s
❯ python -m detection_rules kibana import-rules -o -d custom_rules/rules/
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

6 rule(s) successfully imported
 - 794d2fc0-ecd0-4963-99da-fd587666b80d
 - 222e1b03-fdc9-42a5-911e-2e3e0533549a
 - 230e7043-b0fc-44eb-86b8-e9eb1e5e38d4
 - 09ea590e-69d1-44de-a386-3d84d85cc2f2
 - 69b96ef7-f00f-473b-9525-81123329a036
 - e8e3af2a-11b8-4ab7-9ca1-c6db621ea89d

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 8s
❯ python -m detection_rules import-rules-to-repo /home/forteea1/Code/dac_demo/detection-rules-dac-demo/exports/20240518T115752L.ndjson -s custom_rules/rules --required-only
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for custom_rules/rules/dac_demo_dev_rule_1_updated.toml
[+] Building rule for custom_rules/rules/dac_demo_dev_rule_2_feedback.toml

…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules export-rules-from-repo
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Exported 6 rules into /home/forteea1/Code/clean_mains/detection-rules/exports/20240523T161813L.ndjson

…pport-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte took 2s
❯ python -m detection_rules view-rule custom_rules/rules/command_and_control_dac_demo_dev_rule_1.toml --rule-format
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[metadata]
creation_date = "2024/05/23"
maturity = "production"
updated_date = "2024/05/23"

[rule]
actions = []
author = ["DAC User"]
description = "Test Rule"
enabled = true
exceptions_list = []
false_positives = []
from = "now-9m"
index = ["logs-endpoint.events.*"]
interval = "5m"
language = "eql"
license = "Elastic License v2"
max_signals = 100
name = "DAC Demo Dev Rule 1"
note = "Test Note"
references = ["https://dac-reference.readthedocs.io/en/latest/"]
revision = 0
risk_score = 47
risk_score_mapping = []
rule_id = "52e84641-7483-4a00-a301-13baffa51b5f"
setup = "Test Setup"
severity = "medium"
severity_mapping = []
tags = [
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Command and Control",
    "Data Source: Elastic Defend",
]
timestamp_override = "event.ingested"
to = "now"
type = "eql"
version = 1

query = '''
process where host.os.type == "linux" and process.name == "updated"
'''


[[rule.related_integrations]]
package = "endpoint"
version = "^8.2.0"

[[rule.required_fields]]
ecs = true
name = "host.os.type"
type = "keyword"

[[rule.required_fields]]
ecs = true
name = "process.name"
type = "keyword"

[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1572"
name = "Protocol Tunneling"
reference = "https://attack.mitre.org/techniques/T1572/"


[rule.threat.tactic]
id = "TA0011"
name = "Command and Control"
reference = "https://attack.mitre.org/tactics/TA0011/"



…mited-support-for-version-and-revision-to-baseruledata [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ python -m detection_rules view-rule custom_rules/rules/command_and_control_dac_demo_dev_rule_1.toml --api-format
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "actions": [],
  "author": [
    "DAC User"
  ],
  "description": "Test Rule",
  "enabled": true,
  "exceptions_list": [],
  "false_positives": [],
  "from": "now-9m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "interval": "5m",
  "language": "eql",
  "license": "Elastic License v2",
  "max_signals": 100,
  "name": "DAC Demo Dev Rule 1",
  "note": "Test Note",
  "query": "process where host.os.type == \"linux\" and process.name == \"updated\"\n",
  "references": [
    "https://dac-reference.readthedocs.io/en/latest/"
  ],
  "related_integrations": [
    {
      "package": "endpoint",
      "version": "^8.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "host.os.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    }
  ],
  "revision": 0,
  "risk_score": 47,
  "risk_score_mapping": [],
  "rule_id": "52e84641-7483-4a00-a301-13baffa51b5f",
  "setup": "Test Setup",
  "severity": "medium",
  "severity_mapping": [],
  "tags": [
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Command and Control",
    "Data Source: Elastic Defend"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0011",
        "name": "Command and Control",
        "reference": "https://attack.mitre.org/tactics/TA0011/"
      },
      "technique": [
        {
          "id": "T1572",
          "name": "Protocol Tunneling",
          "reference": "https://attack.mitre.org/techniques/T1572/"
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "to": "now",
  "type": "eql",
  "version": 1
}

@@ -64,6 +64,16 @@ def __getitem__(self, item) -> VersionLockFileEntry:
raise KeyError(item)
return self.data[item]

@classmethod
def from_dict(cls: Type[ClassT], obj: dict) -> ClassT:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we still need these methods?

"""Assemble all the rules into Kibana-ready release files."""
config = load_dump(str(config_file))['package']
if RULES_CONFIG.bypass_version_lock:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for future: we can cleanup all these calls with a bypass_check decorator for applicable functions

Comment on lines 757 to 758
if not deprecation_folder:
deprecation_folder = RULES_CONFIG.rule_dirs[0] / '_deprecated'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to be explicit than implicit - with no guarantee of ordering, it seems somewhat random. This could even change internally, if say we added hunting to the 0 index.

Suggested change
if not deprecation_folder:
deprecation_folder = RULES_CONFIG.rule_dirs[0] / '_deprecated'

There are a few alternative better options:

  • you can use click callbacks (or default as a function) to make the default rule.get_base_rule_dir() / '_deprecated' / rule_file.name if no custom rules dir envvar is set otherwise require
  • update the config to explicitly set a deprecated folder (with comments that its not needed in some instances)
  • just make it an argument or required option always, where even we must explicitly call the deprecated dir

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be done in another PR along with the other examples DEFAULT_PREBUILT_RULES_DIRS[0]

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid this ambiguous situation, we decided the best course of action at this point is to go with the 3rd option and just make it a required click option - enforcing it to always be explicit (including for us)

@brokensound77
Copy link
Collaborator

brokensound77 commented May 24, 2024

I left a few final comments for review.

Beyond that, I think the last remaining big blocker is this comment - around the use of autobumped_version and how the updated functionality does not fit with the intent of the method.

For the sake of progress, we can address it in a future PR, but I think we should def revisit it before declaring alpha

@Mikaayenson
Copy link
Collaborator Author

I left a few final comments for review.

Beyond that, I think the last remaining big blocker is this comment - around the use of autobumped_version and how the updated functionality does not fit with the intent of the method.

For the sake of progress, we can address it in a future PR, but I think we should def revisit it before declaring alpha

I left a few final comments for review.

Beyond that, I think the last remaining big blocker is this comment - around the use of autobumped_version and how the updated functionality does not fit with the intent of the method.

For the sake of progress, we can address it in a future PR, but I think we should def revisit it before declaring alpha

++ Concur I think any more ideas should be migrated to another PR.

traut
traut previously approved these changes May 24, 2024
Copy link
Collaborator

@brokensound77 brokensound77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We were able to discuss the remaining two items in depth (albeit, maybe too long and in depth 😅 )

1. Deprecated folder

To avoid the ambiguity and introducing an ambiguous intermediate state, the code changes suggested changes the deprecated folder to always be explicitly passed

2. latest_version and autobumped_version

After hours of discussion, we realized that, beyond just my concern of autobumped_version functionally changing too far, that implicitly returning 1 without control was also problematic.

We took advantage of this to add more clarity, since the methods and constructs as is were developed and exist in a state where the only versioning concept is version lock. To that end, the summary of requested changes:

  • rename latest_version to saved_version, clarify in the docstring that it pulls saved versions only, with precedence to toml file (over version lock), where None will be returned if not present
    • a HUGE advantage of this change is it now gives us a way to differentiate between defer to kibana and define in rule toml (previously we could only determine bypass or not). This opens up opportunity for tightening down in future PRs (such as, within the rule loader, enforcing a consistent state)
  • update autobumped_version to raise the error when BYPASS_VERSION_LOCK since it will never be valid
  • expose a new convenience method get_synthetic_version with control over how to handle missing (1 or None)
    • there are 8 existing calls using autobumped_version which will need to be changed, but an issue can be made for that to happen in the next PR

Github rendering the diffs is a bit ugly and may be making this less readable, so here is the flattened diff of the 2 changes to help clarify

diff --git a/detection_rules/devtools.py b/detection_rules/devtools.py
index c0b0a1150..386b7dc20 100644
--- a/detection_rules/devtools.py
+++ b/detection_rules/devtools.py
@@ -739,7 +739,8 @@ def search_rule_prs(ctx, no_loop, query, columns, language, token, threads):
 
 @dev_group.command('deprecate-rule')
 @click.argument('rule-file', type=Path)
-@click.option('--deprecation-folder', '-d', type=Path)
+@click.option('--deprecation-folder', '-d', type=Path, required=True,
+              help='Location to move the deprecated rule file to')
 @click.pass_context
 def deprecate_rule(ctx: click.Context, rule_file: Path, deprecation_folder: Path):
     """Deprecate a rule."""
@@ -754,9 +755,6 @@ def deprecate_rule(ctx: click.Context, rule_file: Path, deprecation_folder: Path
         ctx.exit()
 
     today = time.strftime('%Y/%m/%d')
-    if not deprecation_folder:
-        deprecation_folder = RULES_CONFIG.rule_dirs[0] / '_deprecated'
-
     deprecated_path = deprecation_folder / rule_file.name
 
     # create the new rule and save it
diff --git a/detection_rules/packaging.py b/detection_rules/packaging.py
index 050af8b9e..bc0884f0c 100644
--- a/detection_rules/packaging.py
+++ b/detection_rules/packaging.py
@@ -92,10 +92,10 @@ class Package(object):
         self.historical = historical
 
         if min_version is not None:
-            self.rules = self.rules.filter(lambda r: min_version <= r.contents.latest_version)
+            self.rules = self.rules.filter(lambda r: min_version <= r.contents.saved_version)
 
         if max_version is not None:
-            self.rules = self.rules.filter(lambda r: max_version >= r.contents.latest_version)
+            self.rules = self.rules.filter(lambda r: max_version >= r.contents.saved_version)
 
         assert not RULES_CONFIG.bypass_version_lock, "Packaging can not be used when version locking is bypassed."
         self.changed_ids, self.new_ids, self.removed_ids = \
diff --git a/detection_rules/rule.py b/detection_rules/rule.py
index 973bf5a9f..bac68c718 100644
--- a/detection_rules/rule.py
+++ b/detection_rules/rule.py
@@ -937,7 +937,7 @@ class BaseRuleContents(ABC):
         pass
 
     def lock_info(self, bump=True) -> dict:
-        version = self.autobumped_version if bump else (self.latest_version or 1)
+        version = self.autobumped_version if bump else (self.saved_version or 1)
         contents = {"rule_name": self.name, "sha256": self.sha256(), "version": version, "type": self.type}
 
         return contents
@@ -984,12 +984,12 @@ class BaseRuleContents(ABC):
             return max_allowable_version - current_version - 1
 
     @property
-    def latest_version(self) -> Optional[int]:
-        """Retrieve the latest known version of the rule, considering the version lock setting."""
+    def saved_version(self) -> Optional[int]:
+        """Retrieve the version from the version.lock or from the file if version locking is bypassed."""
         toml_version = self.data.get("version")
 
         if BYPASS_VERSION_LOCK:
-            return toml_version or 1
+            return toml_version
 
         if toml_version:
             print(f"WARNING: Rule {self.name} - {self.id} has a version set in the rule TOML."
@@ -1001,10 +1001,10 @@ class BaseRuleContents(ABC):
     @property
     def autobumped_version(self) -> Optional[int]:
         """Retrieve the current version of the rule, accounting for automatic increments."""
-        version = self.latest_version
+        version = self.saved_version
 
         if BYPASS_VERSION_LOCK:
-            return version
+            raise NotImplementedError("This method is not implemented when version locking is not in use.")
 
         # Default to version 1 if no version is set yet
         if version is None:
@@ -1013,6 +1013,14 @@ class BaseRuleContents(ABC):
         # Auto-increment version if the rule is 'dirty' and not bypassing version lock
         return version + 1 if self.is_dirty else version
 
+    def get_synthetic_version(self, use_default: bool) -> Optional[int]:
+        """
+        Get the latest actual representation of a rule's version, where changes are accounted for automatically when
+        version locking is used, otherwise, return the version defined in the rule toml if present else optionally
+        default to 1.
+        """
+        return self.autobumped_version or self.saved_version or (1 if use_default else None)
+
     @classmethod
     def convert_supported_version(cls, stack_version: Optional[str]) -> Version:
         """Convert an optional stack version to the minimum for the lock in the form major.minor."""
diff --git a/detection_rules/version_lock.py b/detection_rules/version_lock.py
index 43d7c1cac..b48515f1c 100644
--- a/detection_rules/version_lock.py
+++ b/detection_rules/version_lock.py
@@ -189,7 +189,7 @@ class VersionLock:
 
         already_deprecated = set(current_deprecated_lock)
         deprecated_rules = set(rules.deprecated.id_map)
-        new_rules = set(rule.id for rule in rules if rule.contents.latest_version is None) - deprecated_rules
+        new_rules = set(rule.id for rule in rules if rule.contents.saved_version is None) - deprecated_rules
         changed_rules = set(rule.id for rule in rules if rule.contents.is_dirty) - deprecated_rules
 
         # manage deprecated rules

If it is easier for me to just push these unstaged changes, let me know


After these 2, this LGTM

@@ -966,17 +985,32 @@ def get_version_space(self) -> Optional[int]:

@property
def latest_version(self) -> Optional[int]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def latest_version(self) -> Optional[int]:
def saved_version(self) -> Optional[int]:

"""Retrieve the latest known version of the rule."""
min_stack = self.get_supported_version()
return self.version_lock.get_locked_version(self.id, min_stack)
"""Retrieve the latest known version of the rule, considering the version lock setting."""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Retrieve the latest known version of the rule, considering the version lock setting."""
"""Retrieve the version from the version.lock or from the file if version locking is bypassed."""

toml_version = self.data.get("version")

if BYPASS_VERSION_LOCK:
return toml_version or 1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return toml_version or 1
return toml_version


@property
def autobumped_version(self) -> Optional[int]:
"""Retrieve the current version of the rule, accounting for automatic increments."""
version = self.latest_version

if BYPASS_VERSION_LOCK:
return version
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return version
raise NotImplementedError("This method is not implemented when version locking is not in use.")

if version is None:
return 1

# Auto-increment version if the rule is 'dirty' and not bypassing version lock
return version + 1 if self.is_dirty else version

@classmethod
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@classmethod
def get_synthetic_version(self, use_default: bool) -> Optional[int]:
"""
Get the latest actual representation of a rule's version, where changes are accounted for automatically when
version locking is used, otherwise, return the version defined in the rule toml if present else optionally
default to 1.
"""
return self.autobumped_version or self.saved_version or (1 if use_default else None)
@classmethod

Comment on lines 757 to 758
if not deprecation_folder:
deprecation_folder = RULES_CONFIG.rule_dirs[0] / '_deprecated'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid this ambiguous situation, we decided the best course of action at this point is to go with the 3rd option and just make it a required click option - enforcing it to always be explicit (including for us)

@@ -721,21 +739,25 @@ def add_github_meta(this_rule: TOMLRule, status: str, original_rule_id: Optional

@dev_group.command('deprecate-rule')
@click.argument('rule-file', type=Path)
@click.option('--deprecation-folder', '-d', type=Path)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@click.option('--deprecation-folder', '-d', type=Path)
@click.option('--deprecation-folder', '-d', type=Path, required=True, help='Location to move the rule file to')

@eric-forte-elastic eric-forte-elastic dismissed stale reviews from traut and themself via 402677e June 1, 2024 02:07
@eric-forte-elastic
Copy link
Collaborator

Applied the above patch 🚀

@eric-forte-elastic
Copy link
Collaborator

eric-forte-elastic commented Jun 1, 2024

Note unit test failure not related to the code change. Locally works fine (I think it may be a dependency that needs upgrading or something) Re-added in latest commit.

Details

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
detections-as-code enhancement New feature or request python Internal python for the repository
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FR][DAC] Add *LIMITED* support for version and revision to BaseRuleData
4 participants