Releases: certtools/intelmq
Bugfix release
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md
Core
__init__
: Resolve absolute path forSTATE_FILE_PATH
variable (resolves..
).intelmq.lib.utils
:- log: Do not raise an exception if logging to neither file nor syslog is requested.
- logging StreamHandler: Colorize all warning and error messages red.
- logging FileHandler: Strip all shell colorizations from the messages (#1436).
intelmq.lib.message
:Message.to_json
: Setsort_keys=True
to get reproducible results.drop_privileges
: Handle situations where the user or groupintelmq
does not exist.
intelmq.lib.pipeline
:Amqp._send
andAmqp._acknowledge
: Log traceback in debug mode in case of errors and necessary re-connections.Amqp._acknowledge
: Reset delivery tag if acknowledge was successful.
Bots
Collectors
intelmq.bots.collectors.misp.collector
:- Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).
Parsers
intelmq.bots.parsers.shadowserver.config
: Add some missing fields for the feedaccessible-rdp
(#1463).intelmq.bots.parsers.shadowserver.parser
:- Feed-detection based on file names: The prefixed date is optional now.
- Feed-detection based on file names: Re-detect feed for every report received (#1493).
Experts
intelmq.bots.experts.national_cert_contact_certat
: Handle empty responses by server (#1467).intelmq.bots.experts.maxmind_geoip
: The scriptupdate-geoip-data
now requires a license key as second parameter because of upstream changes (#1484)).
Outputs
intelmq.bots.outputs.restapi.output
: Fix logging of response body if response status code was not ok.
Documentation
- Remove some hardcoded
/opt/intelmq/
paths from code comments and program outputs.
Packaging
- debian/rules: Only replace
/opt/intelmq/
with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements. - debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.
Tests
lib/test_utils
: Skip some tests on Python 3.4 becausecontextlib.redirect_stdout
andcontextlib.redirect_sterr
are not supported on this version.- Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
tests.bots.parsers.html_table
: Make tests independent of current year.
Tools
intelmqctl upgrade-config
: Fix missing substitution in error message "State file %r is not writable.".
Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.1.1
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md
Configuration
- Default configuration:
- Remove discontinued feed "Feodo Tracker Domains" from default configuration.
- Add "Feodo Tracker Browse" feed to default configuration.
Core
intelmq.lib.pipeline
: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (intelmqctl_rabbitmq_monitoring_url
).intelmq.lib.upgrades
: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.intelmq.lib.utils
:TimeoutHTTPAdapter
: A subclass ofrequests.adapters.HTTPAdapter
with the possibility to set the timeout per adapter.create_request_session_from_bot
: Use theTimeoutHTTPAdapter
with the user-defined timeout. Previously the timeout was not functional.
Bots
Parsers
intelmq.bots.parsers.shadowserver.parser
: Fix logging message if the parameterfeedname
is not present.intelmq.bots.parsers.shodan.parser
: Also add fieldclassification.identifier
('network-scan'
) in minimal mode.intelmq.bots.parsers.spamhaus.parser_cert
: Add support for category'misc'
.intelmq.bots.parsers.cymru.parser_cap_program
:- Add support for phishing events without URL.
- Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
intelmq.bots.parsers.microsoft.parser_bingmurls
:- Save the
Tags
data assource.geolocation.cc
.
- Save the
Experts
intelmq.bots.experts.modify.expert
: Fix bug with setting non-string values (#1460).
Outputs
intelmq.bots.outputs.smtp
:
Documentation
- Feeds:
- Fix configuration of
Feodo Tracker Browse
feed.
- Fix configuration of
- Bots:
- Sieve expert: Document behavior of
!=
with lists.
- Sieve expert: Document behavior of
Tests
- Adaption and extension of the test cases to the changes.
Tools
intelmq.bin.intelmqctl
:- check: Check if running the upgrade function for harmonization is necessary.
- upgrade-config: Run the upgrade function for harmonization.
intelmqctl restart
did throw an error as the message for restarting was not defined (#1465).
Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.1.0
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/UPGRADING.md
Core
intelmq.lib.harmonization
:- Use correct parent classes.
- Add
DateTime.convert
as interface for all existing conversion functions. - add
DateTime.convert_from_format
. - add
DateTime.convert_from_format_midnight
. - add
DateTime.convert_fuzzy
.
intelmq.lib.pipeline
:- Redis: Use single connection client if calling bot is not multithreaded. Gives a small speed advantage.
- Require the bot instance as parameter for all pipeline classes.
- New internal variable
_has_message
to keep the state of the pipeline. - Split receive and acknowledge into public-facing and private methods.
- Add
reject_message
method to the Pipeline class for explicit requeue of messages. - AMQP:
- Make exchange configurable.
- If exchange is set, the queues are not declared, the queue name is for routing used by exchanges.
intelmq.lib.bot
:- Log message after successful bot initialization, no log message anymore for ready pipeline.
- Use existing current message if receive is called and the current message still exists.
- Fix handling of received messaged after a sighup that happend during a blocking receving connection using explicit rejection (#1438).
- New method
_parse_common_parameters
called beforeinit
to parse commonly used argument. Currently supported:extract_files
.
intelmq.lib.test
:- Fix the tests broker by providing the testing pipeline.
intelmq.lib.utils
:unzip
:- new parameter
return_names
to optionally return the file names. - support for zip
- new parameters
try_zip
,try_gzip
andtry_tar
to control which compressions are tried. - rewritten to an iterative approach
- new parameter
- add
file_name_from_response
to extract a file name from a Response object for downloaded files.
intelmq.lib.upgrades
: Addedv210_deprecations
for deprecated parameters.
Harmonization
- Add extra to reports.
Bots
Collectors
intelmq.bots.collectors.http.collector_http
:- More extensive usage of
intelmq.lib.utils.unzip
. - Save the file names in the report if files have been extracted form an archive.
- More extensive usage of
intelmq.bots.collectors.rt.collector_rt
:- Save ticket information/metadata in the extra fields of the report.
- Support for RT 3.8 and RT 4.4.
- New parameters
extract_attachment
andextract_download
for generic archive extraction and consistency. The parameterunzip_attachment
is deprecated.
intelmq.bots.collectors.mail.*
: Save email information/metadata in the extra fields of the report. See the bots documentation for a complete list of provided data.intelmq.bots.collectors.mail.collector_mail_attach
: Check for existence/validity of theattach_regex
parameter.- Use the lib's
unzip
function for uncompressing attachments and use the . intelmq.bots.collectors.mail.collector_mail_url
: Save the file name of the downloaded file asextra.file_name
.
intelmq.bots.collectors.amqp.collector_amqp
: New collector to collect data from (remote) AMQP servers, for bot IntelMQ as well as external data.- use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
Parsers
intelmq.bot.parsers.html_table.parser
:- New parameter "html_parser".
- Use time conversion functions directly from
intelmq.lib.harmonization.DateTime.convert
.
- Limit lxml dependency on 3.4 to < 4.4.0 (incompatibility).
intelmq.bots.parsers.netlab_360.parser
: Add support for hajime scanners.intelmq.bots.parsers.hibp.parser_callback
: A new parser to parse data retrieved from a HIBP Enterprise Subscription.intelmq.bots.parsers.shadowserver.parser
:- Ability to detect the feed base on the reports's field
extra.file_name
, so the parameterfeedname
is no longer required and one configured parser can parse any feed (#1442).
- Ability to detect the feed base on the reports's field
Experts
- Add geohash expert.
intelmq.bot.experts.generic_db_lookup.expert
- new optional parameter
engine
withpostgresql
(default) andsqlite
(new) as possible values.
- new optional parameter
Outputs
- Add
intelmq.bots.outputs.touch.output
. intelmq.bot.outputs.postgresql.output
:- deprecated in favor of
intelmq.bot.outputs.sql.output
- Compatibility shim will be available in the 2.x series.
- deprecated in favor of
intelmq.bot.outputs.sql.output
added generic SQL output bot. Comparted to- new optional parameter
engine
withpostgresql
(default) andsqlite
(new) as possible values.
- new optional parameter
intelmq.bots.outputs.stomp.output
: New parametersmessage_hierarchical_output
,message_jsondict_as_string
,message_with_type
,single_key
.
Documentation
- Feeds:
- Add ViriBack feed.
- Add Have I Been Pwned Enterprise Callback.
intelmq.tests.bots.outputs.amqptopic.test_output
: Added.- Move the documentation of most bots from separate README files to the central Bots.md and feeds.yaml files.
Tests
- Travis:
- Use UTC timezone.
- Tests for
utils.unzip
. - Add a new asset: Zip archive with two files, same as with tar.gz archive.
- Added tests for the Mail Attachment & Mail URL collectors.
- Ignore logging-tests on Python 3.7 temporarily (#1342).
Tools
- intelmqctl:
- Use green and red text color for some interactive output to indicate obvious errors or the absence of them.
- intelmqdump:
- New edit action
v
to modify a message saved in the dump (#1284).
- New edit action
Contrib
- malware name mapping:
- Add support for MISP treat actors data, see it's README for more information.
- And handle empty synonyms in misp's galxies data.
- Move apply-Script to the new EventDB directory
- Add support for MISP treat actors data, see it's README for more information.
- EventDB: Scripts for applying malware name mapping and domain suffixes to an EventDB.
Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.0.2
Install documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/UPGRADING.md
As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.
Core
intelmq.lib.bot.CollectorBot
: Support the deprecated parameterfeed
until version 2.2 as the documentation was not properly updated (#1445).intelmq.lib.bot.Bot
:_dump_message
: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50).
intelmq.lib.upgrades.v202_fixes
- Migration of deprecated parameter
feed
for Collectors. - Ripe expert parameter
query_ripe_stat_ip
was not correctly configured inv110_deprecations
, now usequery_ripe_stat_asn
as default if it does not exist.
- Migration of deprecated parameter
intelmq.lib.upgrades.v110_deprecations
: Fix upgrade of ripe expert configuration.intelmq.lib.bot_debugger
:- Fix handling of empty messages generated by parser when user wanted to show the result by "--show-sent" flag.
- Fix handling of sent messages for bots using the
path_permissive
paramter (#1453).
intelmq.lib.pipeline.Amqp
:- use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
- Reconnect once on sending messages if disconnect detected.
Bots
Collectors
intelmq.bots.collectors.api.collector_api
:- Handle non-existing IO loop in shutdown.
- Close socket on shutdown, fixes reloading.
- Marked as non-threadable.
intelmq.bots.collectors.rt.collector_rt
: Check for matching URLs if noattachment_regex
is given.intelmq.bots.collectors.stomp.collector_stomp
: Handle disconnects by actively reconnecting.
Parsers
intelmq.bots.cymru.parser_cap_program
: Fix parsing of the new$certname_$date.txt
report format (#1443):- Support protocol ICMP.
- Fix error message for unsupported protocols.
- Support fields
destination_port_numbers
,port
. - Support for all proxy types without ports.
- Use Country Code of AS as
source.geolocation.cc
. - Support for 'scanner' and 'spam' categories.
- Handle bogus lines with missing separator.
- Fix bug preventing use of old format after using the new format.
- Handle postfix
(total_count:..)
for destination port numbers.
Experts
intelmq.bots.experts.cymru_whois.expert
: Add optional parameteroverwrite
, current behavior wasTrue
, default if not given isTrue
now, will change toFalse
in 3.0.0 (#1452, #1455).intelmq.bots.experts.modify.expert
: Add optional parameteroverwrite
, current behavior wasTrue
, default if not given isTrue
now, will change toFalse
in 3.0.0 (#1452, #1455).intelmq.bots.experts.reverse_dns.expert
: Add optional parameteroverwrite
, current behavior wasTrue
, default if not given isTrue
now, will change toFalse
in 3.0.0 (#1452, #1455).
Outputs
intelmq.bots.outputs.amqptopic.output
: use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
Packaging
- Rules:
- Exclude intelmqsetup tool in packages
- Include update-rfiprisk-data in packages
Tests
- Tests for
intelmq.lib.upgrades.v202_fixes
. - Tests for
intelmq.lib.upgrades.v110_deprecations
. - Extended tests for
intelmq.bots.parser.cymru.parser_cap_program
.
Tools
- intelmqctl:
- More and more precise logging messages for botnet starting and restarting, enable and disable.
- No error message for disabled bots on botnet reload.
- Fix
upgrade-conf
is state file is empty or not existing. - Use arpgarse's
store_true
action for flags instead ofstore_const
. - If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (#1456).
Contrib
- Check MK Statistics Cronjob:
- Use
statistics_*
parameters. - Make file executable
- Handle None values in
*.temporary.*
keys and treat them as 0.
- Use
- systemd:
- Add
PIDFile
parameter to service file.
- Add
Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.0.1
Install documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/UPGRADING.md
As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.
Core
intelmq.lib.harmonization
:IPAddress
: Remove Scope/Zone IDs for IPv6 addresses in sanitation.- All types: Handle
None
for validation and sanitation gracefully.
intelmq.lib.bot
:__version_info__
:- is now available in the top level module.
- uses integer values now instead of strings for numerical version parts
- Also provide (empty)
ROOT_DIR
for non-pip installations. intelmq.lib.upgrades
: New library fileupgrades
with upgrade functions.intelmq.lib.utils
:- New function
setup_list_logging
for intelmqctl check an possibly others.- Fix return values (#1423).
- New function
version_smaller
for version comparisons. - New function
lazy_int
for version conversions. parse_logline
: Handle thread IDs.log
takes a new argumentlogging_level_stream
for the logging level of the console handler.- New constant
LOG_FORMAT_SIMPLE
, used by intelmqctl. - New function
write_configuration
to write dicts to files in the correct json formatting. - New function
create_request_session_from_bot
.
- New function
intelmq.lib.pipeline
:- AMQP:
- Actually use
source/destination_pipeline_amqp_virtual_host
parameter. - Support for SSL with
source/destination_pipeline_ssl
parameter.
- Actually use
- pipeline base class: add missing dummy methods.
- Add missing return types.
- Redis: Evaluate return parameter of queue/key deletion.
- AMQP:
- Variable
STATE_FILE_PATH
added.
Development
intelmq.bin.intelmq_gen_docs
: For yaml usesafe_load
instead of unsafeload
.
Harmonization
- IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
- TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420).
Bots
Collectors
intelmq.bots.collectors.http.collector_http
: Useutils.create_request_session_from_bot
.intelmq.bots.collectors.http.collector_http_stream
: Useutils.create_request_session_from_bot
and thus fix some retries on connection timeouts.intelmq.bots.collectors.mail.collector_mail_url
: Useutils.create_request_session_from_bot
.intelmq.bots.collectors.microsoft.collector_interflow
: Useutils.create_request_session_from_bot
and thus fix retries on connection timeouts.intelmq.bots.collectors.rt.collector_rt
: Useutils.create_request_session_from_bot
and thus fix retries on connection timeouts.intelmq.bots.collectors.twitter.collector_twitter
: Useutils.create_request_session_from_bot
and thus fix retries on connection timeouts for non-twitter connections.
Parsers
intelmq.bots.parsers.n6.parser_n6stomp
: usemalware-generic
instead ofgeneric-n6-drone
for unknown infected system events.intelmq.bots.parsers.abusech.parser_ip
: Support LastOnline column in feodo feed (#1400) and use it fortime.source
if available.intelmq.bots.parsers.netlab_360.parser
: Detect feeds withhttps://
too.
Experts
intelmq.bots.experts.generic_db_lookup
: Recommend psycopg2-binary package.intelmq.bots.experts.modify.expert
:- Compile regular expressions (all string rules) at initialization, improves the speed.
- Warn about old configuration style deprecation.
intelmq.bots.experts.do_portal.expert
:- Use
utils.create_request_session_from_bot
and thus fix retries on connection timeouts (#1432). - Treat "502 Bad Gateway" as timeout which can be retried.
- Use
intelmq.bots.experts.ripe.expert
: Useutils.create_request_session_from_bot
and thus fix retries on connection timeouts.intelmq.bots.experts.url2fqdn.expert
: Support for IP addresses in hostnames (#1416).intelmq.bots.experts.national_cert_contact_certat.expert
: Useutils.create_request_session_from_bot
and thus fix retries on connection timeouts.
Outputs
intelmq.bots.outputs.postgresql
: Recommend psycopg2-binary package.intelmq.bots.outputs.amqptopic
:- Shutdown: Close connection only if connection exists.
- Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages.
- Gracefully handle unroutable messages and give advice.
- Support for connections without authentication.
- Replace deprecated parameter
type
withexchange_type
forexchange_declare
, supporting pika >= 0.11 (#1425). - New parameters
message_hierarchical_output
,message_with_type
,message_jsondict_as_string
. - New parameter
use_ssl
for SSL connections. - New parameter
single_key
for sending single fields instead of the full event.
intelmq.bots.outputs.mongodb.output
: Support for pymongo >= 3.0.0 (#1063, PR#1421).intelmq.bots.outputs.file
:time.*
field serialization: support for microseconds.intelmq.bots.outputs.mongodb.output
: Support for authentication in pymongo >= 3.5 (#1062).intelmq.bots.outputs.restapi.output
: Useutils.create_request_session_from_bot
and thus fix retries on connection timeouts.
Documentation
- Add certbund-contact to the ecosystem document.
- Rename the IDEA expert to "IDEA Converter".
- Add the new configuration upgrade function to the docs.
- User Guide:
- Clarify on Uninstallation
Packaging
- Do not execute the tcp collector tests during debian and ubuntu builds as they fail there.
Tests
intelmq.lib.test
: Disable statistics for test runs of bots.contrib.malware_name_mapping
: Added tests.- Travis: Also run tests of contrib.
Tools
intelmqsetup
: Only change directory ownerships if necessary.intelmqctl
:- Provide new command
upgrade-conf
to uprade configuration to a newer version.- Makes backups of configurations files on its own.
- Also checks for previously skipped or new functions of older versions and catches up.
- Provides logging level on class layer.
- Fix
-q
flag forintelmqctl list queues
by renaming its alternative name to--non-zero
to avoid a name collision with the global--quiet
parameter. - For console output the string
intelmqctl:
at the beginning of each line is no longer present. check
: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully.- Waits for up to 2 seconds when stopping a bot (#1434).
- Exits early on restart when stopping a bot did not work (#1434).
intelmqctl run process -m
debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged.- Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide.
- Provide new command
Contrib
- logcheck rules: Adapt ignore rule to cover the instance IDs of bot names.
- malware name mapping:
- Ignore lines in mapping starting with '#'.
- Optionally include malpedia data.
- Fix command line parsing for not arguments (#1427).
- bash-completion: Support for
intelmqctl upgrade-config
added.
Known issues
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.0.0
Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0/docs/UPGRADING.md
There are some features considered as beta and marked as such in the documentation, do not use them in production yet.
See also the changelog for 2.0.0.beta1 below.
Configurations
- Defaults: New parameters
statistics_host
,statistics_port
,statistics_databasae
,statistics_password
for statistics redis database (#1402).
Core
- Add more and fix some existing type annotations.
intelmq.lib.bot
:- Use
statistics_*
parameters for bot's statistics (#1402). - Introduce
collector_empty_process
for collectors with an emptyprocess()
method, hardcoded 1s minimum sleep time, preventing endless loops, causing high load (#1364). - Allow to disable multithreading by initialization parameter, used by intelmqctl / the bot debugger (#1403).
- Use
intelmq.lib.pipeline
: redis: OOM can also be low memory, add this to log message (#1405).intelmq.lib.harmonization
: ClassificationType: Update RSIT mapping (#1380):- replace
botnet drone
withinfected-system
- replace
infected system
withinfected-system
- replace
ids alert
withids-alert
- replace
c&c
withc2server
- replace
malware configuration
withmalware-configuration
- sanitize replaces these values on the fly
- replace
- Allow using non-opt/ (LSB) paths with environment variable
INTELMQ_PATHS_NO_OPT
. - Disable/disallow threading for all collectors and some other bots.
Development
- Applied isort to all core files and core-related test files, sorting the imports there (every thing except bots and bots' tests).
Harmonization
- See the Core section for the changes in the allowed values for
classification.type
.
Bots
- Use the new RSIT types in several types, see above
Parsers
intelmq.bots.parsers.spamhaus.parser_cert
: Added support forextortion
events.
Experts
- added
intelmq.bots.experts.do_portal.expert
.
Outputs
intelmq.bots.outputs.elasticsearch.output
: Support for TLS added (#1406).intelmq.bots.outputs.tcp.output
: Support non-intelmq counterparts again. New parametercounterpart_is_intelmq
, see NEWS.md for more information (#1385).
Packaging
- Update IntelMQ path fix patch after
INTELMQ_PATHS_NO_OPT
introduction, provideINTELMQ_PATHS_OPT
environment variable for packaged instances.
Tests
test_conf
: For yaml usesafe_load
instead of unsafeload
.- Travis: Switch distribution from trusty to xenial, adapt scripts.
- Add Python 3.7 to tests.
- Don't use Cerberus 1.3 because of pyeve/cerberus#489
Tools
- intelmqdump: Fix creation of pipeline object by providing a logger.
- intelmqctl: Disable multithreading for interactive runs / the bot debugger (#1403).
Known issues
- tests: capture logging with context manager (#1342)
- pymongo 3.0 deprecates used insert method (#1063)
- pymongo >= 3.5: authentication changes (#1062)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
2.0.0 Beta 1
Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md
There are some features considered as beta and marked as such in the documentation, do not use them in production yet.
- upgraded all files to python3-only syntax, e.g. use
super()
instead ofsuper(..., ...)
in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.
Removals of deprecated code:
- Removed compatibility shim
intelmq.bots.collectors.n6.collector_stomp
, useintelmq.bots.collectors.stomp.collector
instead (see #1124). - Removed compatibility shim
intelmq.bots.parsers.cymru_full_bogons.parser
, useintelmq.bots.parsers.cymru.parser_full_bogons
instead. - Removed compatibility shim handing deprecated parameter
feed
for collectors. Usename
instead. - Removed deprecated and unused method
intelmq.lib.pipeline.Pipeline.sleep
. - Removed support for deprecated parameter
query_ripe_stat
inintelmq.bots.experts.ripe.expert
, usequery_ripe_stat_asn
andquery_ripe_stat_ip
instead (#1291). - Removed deprecated and unused function
intelmq.lib.utils.extract_tar
.
Core
lib/pipeline
:- Allow setting the broker of source and destination independently.
- Support for a new AMQP broker. See User Guide for configuration. (#1179)
lib/bot
:- Dump messages locks the dump file using unix file locks (#574).
- Print idle/rate limit time also in human readable format (#1332).
set_request_parameters
: Use{}
as default proxy value instead ofNone
. Allows updating of existing proxy dictionaries.- Bots drop privileges if they run as root.
- Save statistics on successfully and failed processed messages in the redis database 3.
lib/utils
- Function
unzip
to extract files from gzipped and/or tar-archives. - New class
ListHandler
: new handler for logging purpose which saves the messages in a list. - Add function
seconds_to_human
. - Add function
drop_privileges
. parse_relative
: Strip string before parsing.parse_logline
: Do not convert the timestamps to UTC, leave them as is.
- Function
lib/cache
:- Allow ttl to be None explicitly.
- Overwrite existing cache keys in the database instead of discarding the new data.
lib/bot
:bin/intelmqctl
:
Harmonization
Bots
Collectors
- added
intelmq.bots.parsers.opendxl.collector
(#1265). - added
intelmq.bots.collectors.api
: collecting data using an HTTP API (#123, #1187). - added
intelmq.bots.collectors.rsync
(#1286). intelmq.bots.collectors.http.collector_http
:intelmq.collectors.blueliv.collector_crimeserver
: Allow setting the API URL by parameter (#1336).intelmq.collectors.mail
:- Use internal lib for functionality.
- Add
intelmq.bots.collectors.mail.collector_mail_body
. - Support for
ssl_ca_certificate
parameter (#1362).
Parsers
- added
intelmq.bots.parsers.mcafee.parser_atd
(#1265). intelmq.bots.parsers.generic.parser_csv
:- New parameter
columns_required
to optionally ignore parse errors for columns.
- New parameter
- added
intelmq.bots.parsers.cert_eu.parser_csv
(#1287).- Do not overwrite the local
time.observation
with the data from the feed. The feed's field 'observation time' is now saved in the fieldextra.cert_eu_time_observation
. - Fix parsing of
asn
(renamed tosource asn
,source.asn
internally) and handle existingfeed.accuracy
for parsingconfidence
. - Update columns and mapping to current (2019-04-02) data.
- Do not overwrite the local
- added
intelmq.bots.parsers.surbl.surbl
- added
intelmq.bots.parsers.html_table
(#1381). intelmq.bot.parsers.netlab_360.parser
: Handle empty lines containing blank characters (#1393).intelmq.bots.parsers.n6.parser_n6stomp
: Handle events without IP addresses.intelmq.bots.parsers.cymru.parser_cap_program
: Handle new feed format.intelmq.bots.parsers.shadowserver
:- Add support for the
Accessible-FTP
feed (#1391).
- Add support for the
intelmq.bots.parsers.dataplane.parser
:- Fix parse errors and log more context (#1396).
- added
intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py
andintelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py
(#1373).
Experts
- added
intelmq.bots.experts.recordedfuture_iprisk
(#1267). - added
intelmq.bots.experts.mcafee.expert_mar
(1265). - renamed
intelmq.bots.experts.ripencc_abuse_contact.expert
tointelmq.bots.experts.ripe.expert
, compatibility shim will be removed in version 3.0. intelmq.bots.experts.ripe.expert
:- Use a requests session (#1363).
- Set the requests parameters once per session.
intelmq.bots.experts.maxmind_geoip.expert
: New parameteruse_registered
to use the registered country (#1344).intelmq.bots.experts.filter.expert
: Support for paths (#1208).
Outputs
- added
intelmq.bots.experts.mcafee.output_esm
(1265). - added
intelmq.bots.outputs.blackhole
(#1279). intelmq.bots.outputs.restapi.expert
:- Set the requests parameters once per session.
intelmq.bots.outputs.redis
:- New parameter
hierarchichal_output
(#1388). - New parameter
with_type
.
- New parameter
intelmq.bots.outputs.amqptopic.output
: Compatibility with pika 1.0.0 (#1084, #1394).
Documentation
- added documentation for feeds
- CyberCrime Tracker
- Feodo Tracker Latest
- Feeds: Document abuse.ch URLhaus feed (#1379).
- Install and Upgrading: Use
intelmqsetup
tool.
Packaging
Tests
- Add tests of AMQP broker.
- Travis: Change the ownership of
/opt/intelmq
to the current user.
Tools
intelmqctl check
: Now uses the newListHandler
from utils to handle the logging in JSON output mode.intelmqctl run
: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour.intelmqdump
: Inspecting dumps locks the dump file using unix file locks (#574).intelmqctl
:- After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
intelmqsetup
: New tool for initialize an IntelMQ environment.
Contrib
malware_name_mapping
:- Added the script
apply_mapping_eventdb.py
to apply the mapping to an eventdb. - Possibility to add local rules using the download tool.
- Added the script
check_mk
:- Added scripts for monitoring queues and statistics.
Known issues
- Multi-threaded bots require multiple SIGTERMs (#1403)
- Stats can't be saved with AMQP if redis is password-protected (#1402)
- Update taxonomies to current RSIT and vice-versa (#1380)
- stomp collector bot constantly uses 100% of CPU (#1364)
- tests: capture logging with context manager (#1342)
- Consistent message counter log messages for all kind of bots (#1278)
- pymongo 3.0 deprecates used insert method (#1063)
- pymongo >= 3.5: authentication changes (#1062)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
Bugfix release 1.2.0
Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.2/docs/UPGRADING.md
Core
intelmq.lib.bot
:Bot.__handle_sighup
: Handle exceptions inshutdown
method of bots.
Harmonization
- FQDN: Disallow
:
in FQDN values to prevent values like '10.0.0.1:8080' (#1235).
Bots
Collectors
intelmq.bots.collectors.stomp.collector
- Fix name of shutdown method, was ineffective in the past.
- Ignore
NotConnectedException
errors on disconnect during shutdown.
intelmq.bots.collectors.mail.collector_mail_url
: Decode body if it is bytes (#1367).intelmq.bots.collectors.tcp.collector
: Timeout added. More stable version.
Parsers
intelmq.bots.parsers.shadowserver
:intelmq.bots.parsers.microsoft.parser_ctip
:- Workaround for mis-formatted data in
networkdestinationipv4
field (since 2019-03-14). - Ignore "hostname" ("destination.fqdn") if it contains invalid data.
- Workaround for mis-formatted data in
intelmq.bots.parsers.shodan.parser
:- In
minimal_mode
:- Fix the parsing, previously only
source.geolocation.cc
andextra.shodan
was correctly filled with information. - Add a
classification.type
= 'other' to all events. - Added tests for this mode.
- Fix the parsing, previously only
- Normal mode:
- Fix the parsing of
timestamp
to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception. - ISAKMP: Ignore
isakmp.aggressive
, as the content is same asisakmp
or less.
- Fix the parsing of
- In
intelmq.bots.parsers.abusech.parser_ip
: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed.intelmq.bots.parsers.n6.parser
:- Add parsing for fields "confidence", "expires" and "source".
- Add support for type "bl-other" (category "other").
Experts
intelmq.bots.experts.sieve.expert
: Fix key definition to allow field names with numbers (malware.hash.md5
/sha1
, #1371).
Outputs
intelmq.bots.outputs.tcp.output
: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability.
Documentation
- Install: Update operating system versions
- Sieve Expert: Fix
elsif
->elif
. - Rephrase the description of
time.*
fields. - Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued.
Packaging
Tests
- Add missing
__init__.py
files in 4 bot's test directories. Previously these tests have never been executed. intelmq.lib.test
: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g.TestShodanParserBot_minimal
.
Tools
- intelmqctl:
- status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was
None
). - Use logging level from defauls configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally.
- status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was
Known issues
Bugfix release 1.1.1
Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md
Core
lib/harmonization.py
: Changeparse_utc_isoformat
ofDateTime
class from private to public (related to #1322).lib/utils.py
: Add new functionobject_pair_hook_bots
.lib.bot.py
:ParserBot
's methodrecover_line_csv
now also handles giventempdata
.Bot.acknowledge_message()
deletes__current_message
to free the memory, saves memory in idling parsers with big reports.start()
: Warn once per run iferror_dump_message
is set to false.Bot.start()
,ParserBot.process()
: If errors happen on bots without destination pipeline, theon_error
path has been queried and lead to an exception being raised.start()
: Iferror_procedure
is pass and on pipeline errors, the bot retries forever (#1333).
lib/message.py
:lib/pipeline.py
(Redis.receive
): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334).
Default configuration
- Set
error_dump_message
to true by default indefaults.conf
. - Fixed typo in
defaults.conf
:proccess_manager
->process_manager
Development
bin/rewrite_config_files.py
: Fix ordering of BOTS file (#1327).
Harmonization
Update to 2018-09-26 version. New values are per taxonomy:
- Taxonomy 'intrusions':
- "application-compromise"
- "burglary"
- "privileged-account-compromise"
- "unprivileged-account-compromise"
- Taxonomy 'fraud':
- "copyright"
- "masquerade"
- "unauthorized-use-of-resources"
- Taxonomy 'information content security':
- "data-loss"
- Taxonomy 'vulnerable':
- "ddos-amplifier"
- "information-disclosure"
- "potentially-unwanted-accessible"
- "vulnerable-system"
- "weak-crypto"
- Taxonomy 'availability':
- "dos"
- "outage"
- "sabotage"
- Taxonomy 'abusive-content':
- "harmful-speech"
- "violence"
- Taxonomy 'malicious code':
- "malware-distribution"
- Taxonomy 'information-gathering':
- "social-engineering"
- "sniffing"
- Taxonomy 'information content security':
- "Unauthorised-information-access"
- "Unauthorised-information-modification"
Bots
Collectors
intelmq.bots.collectors.http.collector_http
:- Fix parameter name
extract_files
in BOTS (#1331). - Fix handling of
extract_files
parameter if the value is an empty string. - Handle not installed dependency library
requests
gracefully. - Explain
extract_files
parameter in docs and use a sane default in BOTS file.
- Fix parameter name
intelmq.bots.collectors.mail.collector_mail_url
:- Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling.
- Handle HTTP errors (bad status code and timeouts) with
error_procedure
== 'pass' but marking the mail as read and logging the error. - Handle not installed dependency library
requests
gracefully.
intelmq.bots.collectors.http.collector_http_stream
:- Handle not installed dependency library
requests
gracefully.
- Handle not installed dependency library
intelmq.bots.collectors.microsoft.collector_interflow
:- Handle not installed dependency library
requests
gracefully.
- Handle not installed dependency library
intelmq.bots.collectors.rt.collector_rt
:- Handle not installed dependency library
requests
gracefully.
- Handle not installed dependency library
- added
intelmq.bots.collectors.shodan.collector_stream
for collecting shodan stream data (#1096).- Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers.
intelmq.bots.collectors.microsoft.collector_interflow
:- Add check if Cache's TTL is big enough compared to
not_older_than
and throw an error otherwise.
- Add check if Cache's TTL is big enough compared to
Parsers
intelmq.bots.parsers.misp
: Fix Object attribute (#1318).intelmq.bots.parsers.cymru.parser_cap_program
:- Add support for new format (extra data about botnet of 'bots').
- Handle AS number 0.
intelmq.bots.parsers.shadowserver
:- Spam URL reports: remove
src_naics
,src_sic
columns. - fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271).
- Add support in parser to ignore some columns in config file by using
False
as intelmq key. - Add support for the
Outdated-DNSSEC-Key
andOutdated-DNSSEC-Key-IPv6
feeds. - Add support for the
Accessible-Rsync
feed. - Document support for the
Open-LDAP-TCP
feed. - Add support for
Accessible-HTTP
andOpen-DB2-Discovery-Service
(#1349). - Add support for
Accessible-AFP
(#1351). - Add support for
Darknet
(#1353).
- Spam URL reports: remove
intelmq.bots.parsers.generic.parser_csv
: If theskip_header
parameter was set toTrue
, the header was not part of theraw
field as returned by therecover_line
method. The header is now saved and handled correctly by the fixed recovery method.intelmq.bots.parsers.cleanmx.parser
: Use fieldfirst
instead offirsttime
fortime.source
(#1329, #1348).intelmq.bots.parsers.twitter.parser
: Support forurl-normalize
>= 1.4.1 and recommend it. Added new optional parameterdefault_scheme
, passed tourl-normalize
(#1356).
Experts
intelmq.bots.experts.national_cert_contact_certat.expert
:- Handle not installed dependency library
requests
gracefully.
- Handle not installed dependency library
intelmq.bots.experts.ripencc_abuse_contact.expert
:- Handle not installed dependency library
requests
gracefully.
- Handle not installed dependency library
intelmq.bots.experts.sieve.expert
:intelmq.bots.experts.idea.expert
: Add mappings for new harmonizationclassification.type
values, see above.
Outputs
intelmq.bots.outputs.redis
:intelmq.bots.outputs.mongodb
:intelmq.bots.outputs.restapi.output
:- Handle not installed dependency library
requests
gracefully.
- Handle not installed dependency library
Documentation
- FAQ
- Explanation and solution on orphaned queues.
- Section on how and why to remove
raw
data.
- Add or fix the tables of contents for all documentation files.
- Feeds:
- Add SECURITY.md file.
Packaging
- Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).
Tests
intelmq.tests.lib.test_bot
: Skiptest_logging_level_other
on python 3.7 because of unclear behavior related to copies of loggers (#1269).intelmq.tests.bots.collectors.rt.test_collector
: Remove test because the REST interface of the instance has been closed (see also python-rt/python-rt#28).
Tools
intelmqctl check
: Shows more detailed information on orphaned queues.intelmqctl
:- Correctly determine the status of bots started with
intelmqctl run
. - Fix output of errors during bot status determination, making it compatible to IntelMQ Manager.
check
subcommand: Show bot ID for messages also in JSON output.run [bot-id] process -m [message]
works also with bots without a configured source pipeline (#1307).
- Correctly determine the status of bots started with
Contrib
- elasticsearch/elasticmapper: Add tlp field (#1308).
feeds-config-generator/intelmq_gen_feeds_conf
:
Known issues
Feature release 1.1.0
Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.0/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.0/docs/UPGRADING.md
- Support for Python 3.3 has been dropped in IntelMQ and some dependencies of it. Python 3.3 reached its end of life and Python 3.4 or newer is a hard requirement now.
- The list of feeds docs/Feeds.md has now a machine-readable equivalent YAML file in intelmq/etc/feeds.yaml
A tool to convert from yaml to md has been added.
Tools
intelmq_gen_feeds_docs
addded to bin directory, allows generating the Feeds.md documentation file from feeds.yamlintelmq_gen_docs
merges bothintelmq_gen_feeds_docs
andintelmq_gen_harm_docs
in one file and automatically updates the documentation files.
intelmqctl
intelmqctl start
prints the bot's last error messages if the bot failed to start (#1021).intelmqctl start
message "is running" is printed every time. (Until now, it wasn't said when a bot was just starting.)intelmqctl start/stop/restart/reload/status
now has a "--group" flag which allows you to specify the group of the bots that should be influenced by the command.intelmqctl check
checks for defaults.conf completeness if the shipped file from the package can be found.intelmqctl check
shows errors for non-importable bots.intelmqctl list bots -q
only prints the IDs of enabled bots.intelmqctl list queues-and-status
prints both queues and bots statuses (so that it can be used in eg. intelmq-manager).intelmqctl run
parameter for showing a sent message.intelmqctl run
if message is sent to a non-default path, it is printed out.intelmqctl restart
bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226).intelmqctl check
: New parameter--no-connections
to prevent the command from making connections e.g. to the redis pipeline.sintelmqctl list queues
: don't display named paths amongst standard queues.- The process status test failed if the PATH did not include the bot executables and the
which
command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297).
Contrib
- tool
feeds-config-generator
to automatically generate the collector and parser runtime and pipeline configurations. malware_name_mapping
: Download and convert tool for malware family name mapping has been added.- Added a systemd script which creates systemd units for bots (#953).
contrib/cron-jobs/update-asn-data
,contrib/cron-jobs/update-geoip-data
,contrib/cron-jobs/update-tor-nodes
: Errors produce proper output.
Core
- lib/bot
- use SIGTERM instead of SIGINT to stop bots (#981).
- Bots can specify a static method
check(parameters)
which can perform individual checks specific to the bot.
These functions will be called byintelmqctl check
if the bot is configured with the given parameters - top level bot parameters (description, group, module, name) are exposed as members of the class.
- The parameter
feed
for collectors is deprecated for 2.0 and has been replaced by the more consistentname
(#1144). - bug: allow path parameter for CollectorBot class.
- Handle errors better when the logger could not be initialized.
ParserBot
:- For the csv parsing methods,
ParserBot.csv_params
is now used for all these methods. ParserBot.parse_csv_dict
now saves the field names inParserBot.csv_fieldnames
.ParserBot.parse_csv_dict
now saves the raw current line inParserBot.current_line
.ParserBot.recover_line_csv_dict
now uses the raw current line.
- For the csv parsing methods,
- lib/message:
- Subitems in fields of type
JSONDict
(see below) can be accessed directly. E.g. you can do:
event['extra.foo'] = 'bar'
event['extra.foo'] # gives 'bar'
It is still possible to set and get the field as whole, however this may be removed or changed in the future:
event['extra'] = '{"foo": "bar"}'
event['extra'] # gives '{"foo": "bar"}'
"Old" bots and configurations compatible with 1.0.x do still work.
Also, the extra field is now properly exploded when exporting events, analogous to all other fields.
Thein
operator works now for both - the old and the new - behavior. Message.add
: The parameteroverwrite
accepts now three different values:True
,False
andNone
(new).
True: An existing value will be overwritten
False: An existing value will not be overwritten (previously an exception has been raised when the value was given).
None (default): If the value exists anKeyExists
exception is thrown (previously the same as False).
This allows shorter code in the bots, as an 'overwrite' configuration parameter can be directly passed to the function.- The message class has now the possibility to return a default value for non-exisiting fields, see
Message.set_default_value
. - Message.get behaves the same like
Message.__getitem__
(#1305).
- Subitems in fields of type
- Add
RewindableFileHandle
to utils making handling of CSV files more easy (optionally) - lib/pipeline:
lib/harmonization
: AcceptAS
prefix for ASN values (automatically stripped).
Bots
- Removed print statements from various bots.
- Replaced various occurences of
self.logger.error()
+self.stop()
withraise ValueError
.
Collectors
bots.collectors.mail
:- New parameters;
sent_from
: filter messages by sender,sent_to
: filter messages by recipient - More debug logs
- New parameters;
bots.collectors.n6.collector_stomp
: renamed tobots.collectors.stomp.collector
(#716)- bots.collectors.rt:
- New parameter
search_requestor
to search for field Requestor. - Empty strings and
null
as value for search parameters are ignored. - Empty parameters
attachment_regex
andurl_regex
handled.
- New parameter
bots.collectors.http.collector_http
: Ability to optionally use the current time in parameterhttp_url
, added parameterhttp_url_formatting
.bots.collectors.stomp.collector
: Heartbeat timeout is now logged with log level info instead of warning.- added
intelmq.bots.collectors.twitter.collector_twitter
- added
intelmq.bots.collectors.tcp.collector
that can be bound to another IntelMQ instance by a TCP output bots.collectors.microsoft.collector_interflow
: added for MS interflow API- Automatic ungzipping for .gz files.
- added
intelmq.bots.collectors.calidog.collector_certstream
for collecting certstream data (#1120). - added
intelmq.bots.collectors.shodan.collector_stream
for collecting shodan stream data (#1096).- Add proxy support.
- Fix handling of parameter
countries
.
Parsers
bots.parsers.shadowserver
:- changed feednames . Please refer to it's README for the exact changes.
- If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.
Previously errors like these were only logged and ignored otherwise. - add support for the feeds
- Remove deprecated parameter
override
, useoverwrite
instead (#1071). - The
raw
values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011).
- The Generic CSV Parser
bots.parsers.generic.parser_csv
:- It is possible to filter the data before processing them using the new parameters
filter_type
andfilter_text
. - It is possible to specify multiple columns using
|
character in parametercolumns
. - The parameter
time_format
now supports'epoch_millis'
for seconds since the Epoch, milliseconds are supported but not used.
- It is possible to filter the data before processing them using the new parameters
- renamed
bots.parsers.cymru_full_bogons.parser
tobots.parsers.cymru.parser_full_bogons
, compatibility shim will be removed in version 2.0 - added
bots.parsers.cymru.parser_cap_program
- added
intelmq.bots.parsers.zoneh.parser
for ZoneH feeds - added
intelmq.bots.parsers.sucuri.parser
- added
intelmq.bots.parsers.malwareurl.parser
- added
intelmq.bots.parsers.threatminer.parser
- added
intelmq.bots.parsers.webinspektor.parser
- added
intelmq.bots.parsers.twitter.parser
- added
intelmq.bots.parsers.microsoft.parser_ctip
- ignore the invalid IP '0.0.0.0' for the destination
- fix the raw/dumped messages, did not contain the paling list previously.
- use the new harmonization field
tlp
instead ofextra.tlp
.
bots.parsers.alienvault.parser_otx
: Save TLP data in the new harmonization fieldtlp
.- added
intelmq.bots.parsers.openphish.parser_commercial
- added
intelmq.bots.parsers.microsoft.parser_bingmurls
- added
intelmq.bots.parsers.calidog.parser_certstream
for parsing certstream data (#1120). - added
intelmq.bots.parsers.shodan.parser
for parsing shodan data (#1096). - change the classification type from 'botnet drone' to infected system' in various parses.
intelmq.bots.parsers.spamhaus.parser_cert
: Added support for all known bot types.
Experts
- Added sieve expert for filtering and modifying events (#1083)
- capable of distributing the event to appropriate named queues
bots.experts.modify
- default rulesets: all malware name mappings have been migrated to the Malware Name Mapping repository ruleset. See the new added contrib tool for download and conversion.
- new parameter
case_sensitive
(default: True)
- Added wait expert for sleeping
- A...