-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modify AWS module to process logs from WAF v2 #22572
Comments
Update
|
Update
|
Update
{"timestamp":"2024-04-29T11:38:31.092+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"80442","firedtimes":3,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxx.822910","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-7xxd1f-bfed-4b00-9f5e-88ce44718194","s3bucket":"wazuh-aws-wodle-waf"},"timestamp":"1576280412771.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:ap-southeast-2:1xxx5:regional/webacl/test/111","terminatingRuleId":"STMTest_SQLi_XSS","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":{"conditionType":"SQL_INJECTION","location":"UNKNOWN","matchedData":["10","AND","1"]},"httpSourceName":"ALB","httpSourceId":"alb","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"httpRequest":{"clientIp":"1.1.1.1","country":"AU","httpVersion":"HTTP/1.1","httpMethod":"POST","requestId":"null"},"labels":{"name":"value"},"source":"waf"}},"location":"Wazuh-AWS"}
|
Update
root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Marker: 2019/10/22
DEBUG: +++ No logs to process in bucket: aws-waf-logs-wodle-v2
DEBUG: +++ DB Maintenance |
UpdateI have been setting up a real WAF environment for log generation in the S3 bucket:
|
Update
{"timestamp":1714735190413,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-east-1:xxxxxx:regional/webacl/AWS-WAF-V2/xxxxxxx817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"xxxx422-app/ABL-WAF-V2/xxxxxc1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":402,"httpRequest":{"clientIp":"46.174.191.28","country":"UA","headers":[{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko"},{"name":"Accept","value":"*/*"}],"uri":"/","args":"","httpVersion":"HTTP/1.0","httpMethod":"GET","requestId":"xxxxxxxx0713ede17e"}}
|
Update
wazuh/wodles/aws/buckets_s3/guardduty.py Lines 12 to 17 in 1e51c2d
|
Update
root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/2024/05/06
DEBUG: +++ Unexpected error: 'bucket'
ERROR: Unexpected error querying/working with objects in S3: 'bucket' I've been testing changes in |
Update
|
UpdateThe necessary changes were made to obtain records of WAF native :root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2019/10/22
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
DEBUG: +++ DB Maintenance As you can see, the logs belonging to WAF v2 routes were received. Next, the output of these logs has been verified at:
root@wazuh-master:/# grep '567970947422' /var/ossec/logs/alerts/alerts.log
{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz", "s3bucket": "aws-waf-logs-wodle-v2"}, "timestamp": 1714734687613, "formatVersion": 1, "webaclId": "arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-xxxxxx", "terminatingRuleId": "Dont-allow-HTTP-GET-and-POST", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [], "httpSourceName": "ALB", "httpSourceId": "567970947422-app/ABL-WAF-V2/xxxxx", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [], "responseCodeSent": 402, "httpRequest": {"clientIp": "87.236.176.171", "country": "GB", "headers": {"Host": "35.153.251.153", "User-Agent": "Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)", "Connection": "close", "Accept": "*/*", "Accept-Encoding": "gzip"}, "uri": "/", "args": "", "httpVersion": "HTTP/1.1", "httpMethod": "GET", "requestId": "1-6634c65f-xxxxxxxxx"}, "source": "waf"}}
aws.log_info.log_file: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz", "s3bucket": "aws-waf-logs-wodle-v2"}, "timestamp": 1714735190413, "formatVersion": 1, "webaclId": "arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857", "terminatingRuleId": "Dont-allow-HTTP-GET-and-POST", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [], "httpSourceName": "ALB", "httpSourceId": "567970947422-app/ABL-WAF-V2/xxxxxxx", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [], "responseCodeSent": 402, "httpRequest": {"clientIp": "46.174.191.28", "country": "UA", "headers": {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko", "Accept": "*/*"}, "uri": "/", "args": "", "httpVersion": "HTTP/1.0", "httpMethod": "GET", "requestId": "xxxxxxxxxxxxxxxx"}, "source": "waf"}}
aws.log_info.log_file: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
{"timestamp":"2024-05-13T11:22:26.600+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"xxx","firedtimes":1,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxxxxx","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz","s3bucket":"aws-waf-logs-wodle-v2"},"timestamp":"1714734687613.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"567970947422-app/ABL-WAF-V2/27426a40f4ac1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"responseCodeSent":"402","httpRequest":{"clientIp":"87.236.176.171","country":"GB","headers":{"Host":"35.153.251.153","User-Agent":"Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)","Connection":"close","Accept":"*/*","Accept-Encoding":"gzip"},"uri":"/","httpVersion":"HTTP/1.1","httpMethod":"GET","requestId":"1xxxxxxxxx"},"source":"waf"}},"location":"Wazuh-AWS"}
{"timestamp":"2024-05-13T11:22:26.885+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"xxxx","firedtimes":2,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxxxxx","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz","s3bucket":"aws-waf-logs-wodle-v2"},"timestamp":"1714735190413.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"567970947422-app/ABL-WAF-V2/27426a40f4ac1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"responseCodeSent":"402","httpRequest":{"clientIp":"46.174.191.28","country":"UA","headers":{"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko","Accept":"*/*"},"uri":"/","httpVersion":"HTTP/1.0","httpMethod":"GET","requestId":"xxxxxxxx"},"source":"waf"}},"location":"Wazuh-AWS"} WAF kinesis :The operation of WAF Kinesis was also verified, including the deprecation message starting from version root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket wazuh-aws-wodle-waf --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
The functionality to process WAF logs stored in S3 via Kinesis was deprecated in 5.0. Consider configuring WAF to store its logs directly in an S3 bucket instead. Check https://documentation.wazuh.com/current/amazon/services/supported-services/waf.html for more information.
DEBUG: +++ Marker: 2019/10/22
DEBUG: ++ Found new log: 2019/10/23/10/aws-waf-logs-delivery-stream-1-2019-10-23-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-1
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-2
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-3
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-multiple-values-in-ruleGroupList
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-multiple-values-in-ruleGroupList
DEBUG: +++ DB Maintenance
|
UpdateThe tests related to (unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests/test_waf.py -v
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0 -- /home/wazuh/venv/unittest-env/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.10.12', 'Platform': 'Linux-6.5.0-17-generic-x86_64-with-glibc2.35', 'Packages': {'pytest': '7.3.1', 'pluggy': '1.4.0'}, 'Plugins': {'anyio': '4.3.0', 'aiohttp': '1.0.4', 'trio': '0.8.0', 'html': '2.1.1', 'metadata': '3.1.0', 'asyncio': '0.18.1', 'tavern': '1.23.5'}}
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 15 items
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_initializes_properly PASSED [ 6%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-False] PASSED [ 13%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-True] PASSED [ 20%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-invalid-json-True] PASSED [ 26%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-wrong-structure-True] PASSED [ 33%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file_handles_exception_on_invalid_argument[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-invalid-json-False-SystemExit] PASSED [ 40%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file_handles_exception_on_invalid_argument[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-wrong-structure-False-SystemExit] PASSED [ 46%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type[object_list0-True] PASSED [ 53%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type[object_list1-False] PASSED [ 60%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type_handles_exceptions PASSED [ 66%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_service_prefix PASSED [ 73%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_base_prefix[True] PASSED [ 80%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_base_prefix[False] PASSED [ 86%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_iter_regions_and_accounts[True] PASSED [ 93%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_iter_regions_and_accounts[False] PASSED [100%]
=============================================================================================== 15 passed in 0.27s ================================================================================================
(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 598 items
wodles/aws/tests/test_aws_bucket.py ....................................................................................................................................................................... [ 27%]
.................................... [ 33%]
wodles/aws/tests/test_aws_s3.py .................... [ 37%]
wodles/aws/tests/test_aws_service.py .... [ 37%]
wodles/aws/tests/test_cloudtrail.py .. [ 38%]
wodles/aws/tests/test_cloudwatchlogs.py ..................................................... [ 47%]
wodles/aws/tests/test_config.py .............................................................................. [ 60%]
wodles/aws/tests/test_guardduty.py ................. [ 63%]
wodles/aws/tests/test_inspector.py ...... [ 64%]
wodles/aws/tests/test_load_balancers.py ............ [ 66%]
wodles/aws/tests/test_s3_log_handler.py ................ [ 68%]
wodles/aws/tests/test_server_access.py ................................. [ 74%]
wodles/aws/tests/test_sqs_message_processor.py ........ [ 75%]
wodles/aws/tests/test_sqs_queue.py ....... [ 76%]
wodles/aws/tests/test_tools.py .................................. [ 82%]
wodles/aws/tests/test_umbrella.py ...... [ 83%]
wodles/aws/tests/test_vpcflow.py ..................... [ 86%]
wodles/aws/tests/test_waf.py ............... [ 89%]
wodles/aws/tests/test_wazuh_integration.py ............................................................... [100%]
=============================================================================================== 598 passed in 2.94s =============================================================================================== |
UpdateThe requested changes were checked and the tests for root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/None/AWS-WAF-V2/2019/10/22
DEBUG: +++ No logs to process in bucket: aws-waf-logs-wodle-v2
DEBUG: +++ DB Maintenanc As can be seen in the bucket, it is marking the region being iterated over as None. This issue arose due to the change introduced in the review #23397 (comment), without realizing that depending on the type of This was corrected back to how it was before, the other requested changes were added, and the corresponding tests were updated: Tests performed:Tests performedI run the WAF v2 bucket: root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2019/10/22
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
DEBUG: +++ DB Maintenance I run the bucket again after processing the logs: root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
DEBUG: +++ No logs to process in bucket: 567970947422/us-east-1
DEBUG: +++ DB Maintenance I run the Kinesis bucket: root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket wazuh-aws-wodle-waf --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
The functionality to process WAF logs stored in S3 via Kinesis was deprecated in 5.0. Consider configuring WAF to store its logs directly in an S3 bucket instead. Check https://documentation.wazuh.com/current/amazon/services/supported-services/waf.html for more information.
DEBUG: +++ Marker: 2019/10/22
DEBUG: ++ Found new log: 2019/10/23/10/aws-waf-logs-delivery-stream-1-2019-10-23-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake
ERROR: the 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake file doesn't have the expected structure.
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-1
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-2
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-3
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-multiple-values-in-ruleGroupList
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-multiple-values-in-ruleGroupList
DEBUG: +++ DB Maintenance The alerts generated from the logs were also verified. The related tests were checked: Tests(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests/test_waf.py
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 15 items
wodles/aws/tests/test_waf.py ............... [100%]
=============================================================================================== 15 passed in 0.26s ================================================================================================
(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 598 items
wodles/aws/tests/test_aws_bucket.py ....................................................................................................................................................................... [ 27%]
.................................... [ 33%]
wodles/aws/tests/test_aws_s3.py .................... [ 37%]
wodles/aws/tests/test_aws_service.py .... [ 37%]
wodles/aws/tests/test_cloudtrail.py .. [ 38%]
wodles/aws/tests/test_cloudwatchlogs.py ..................................................... [ 47%]
wodles/aws/tests/test_config.py .............................................................................. [ 60%]
wodles/aws/tests/test_guardduty.py ................. [ 63%]
wodles/aws/tests/test_inspector.py ...... [ 64%]
wodles/aws/tests/test_load_balancers.py ............ [ 66%]
wodles/aws/tests/test_s3_log_handler.py ................ [ 68%]
wodles/aws/tests/test_server_access.py ................................. [ 74%]
wodles/aws/tests/test_sqs_message_processor.py ........ [ 75%]
wodles/aws/tests/test_sqs_queue.py ....... [ 76%]
wodles/aws/tests/test_tools.py .................................. [ 82%]
wodles/aws/tests/test_umbrella.py ...... [ 83%]
wodles/aws/tests/test_vpcflow.py ..................... [ 86%]
wodles/aws/tests/test_waf.py ............... [ 89%]
wodles/aws/tests/test_wazuh_integration.py ............................................................... [100%]
=============================================================================================== 598 passed in 2.59s ===============================================================================================
|
UpdateAbout the last requested changes:
|
Description
Our current integration with AWS WAF is only extracting records through Kinesis.
In this issue, we should perform the modifications necessary to our AWS module to fetch the records being generated on AWS WAF v2, following the path
Note
The Migrating your AWS WAF Classic resources to AWS WAF guide might be useful.
Checks
The following elements have been updated or reviewed (should also be checked if no modification is required):
api/test/integration/mapping/_test_mapping.py
).The text was updated successfully, but these errors were encountered: