3.3.0

3.3.0 (2024-03-01)

Configuration

• Add new optional configuration parameters for intelmq.bots.collectors.stomp.collector
  and intelmq.bots.outputs.stomp.output (PR#2408 by Jan Kaliszewski):
  • auth_by_ssl_client_certificate (Boolean, default: true; if false then
    ssl_client_certificate and ssl_client_certificate_key will be ignored);
  • username (STOMP authentication login, default: "guest"; to be used only
    if auth_by_ssl_client_certificate is false);
  • password (STOMP authentication passcode, default: "guest"; to be used only
    if auth_by_ssl_client_certificate is false).
• Add the possibility to set the ssl_ca_certificate configuration parameter for
  intelmq.bots.collectors.stomp.collector and/or intelmq.bots.outputs.stomp.output
  to an empty string - which means that the SSL machinery used for STOMP communication
  will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).

Core

• intelmq.lib.message: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).
• intelmq.lib.exceptions.InvalidKey: Add optional parameter additional_text (PR#2398 by Sebastian Wagner).
• Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
• intelmq.lib.mixins: Add a new class, StompMixin (defined in a new submodule: stomp),
  which provides certain common STOMP-bot-specific operations, factored out from
  intelmq.bots.collectors.stomp.collector and intelmq.bots.outputs.stomp.output
  (PR#2408 and PR#2414 by Jan Kaliszewski).
• intelmq.lib.upgrades: Replace deprecated instances of url2fqdn experts by the new url expert in runtime configuration (PR#2432 by Sebastian Wagner).
• intelmq.lib.bot: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).
• AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
• Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).

Development

• Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).

Data Format

Bots

Collectors

• intelmq.bots.collectors.stomp.collector (PR#2408 and PR#2414 by Jan Kaliszewski):
  • Drop support for versions of stomp.py older than 4.1.12.
  • Update the code to support new versions of stomp.py, including the latest (8.1.0);
    fixes #2342.
  • Add support for authentication based on STOMP login and passcode, introducing three
    new configuration parameters (see above: Configuration).
  • Add support for loading the system’s default CA certificates, as an alternative to
    specifying the CA certificate(s) file path explicitly (see above: Configuration).
  • Fix (by carefully targeted monkey patching) certain security problems caused by
    SSL-related weaknesses that some versions of stomp.py suffer from.
  • Fix the reconnection behavior: do not attempt to reconnect after shutdown. Also,
    never attempt to reconnect if the version of stomp.py is older than 4.1.21 (it
    did not work properly anyway).
  • Add coercion of the port config parameter to int.
  • Add implementation of the check hook (verifying, in particular, accessibility
    of necessary file(s)).
  • Remove undocumented and unused attributes of StompCollectorBot instances:
    ssl_ca_cert, ssl_cl_cert, ssl_cl_cert_key.
    • Minor fixes/improvements and some refactoring (see also above: Core...).
• intelmq.bots.collectors.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
• intelmq.bots.collectors.shadowserver.collector_reports_api:
  • The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
• intelmq.bots.collectors.alienvault_otx.collector (PR#2449 by qux-bbb):
  • Fix modified_pulses_only is always False.

Parsers

• intelmq.bots.parsers.netlab_360.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
• intelmq.bots.parsers.webinspektor.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
• intelmq.bots.parsers.sucuri.parser: Removed as the feed is discontinued. (#2442 by Filip Pokorný)
• intelmq.bots.parsers.shadowserver._config:
  • Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
• intelmq.bots.parsers.cymru: Save current line. (PR by Kamil Mankowski)

Experts

• intelmq.bots.experts.jinja (PR#2417 by Mikk Margus Möll):
  • Add optional socket_perms and socket_group parameters to change
    file permissions on socket file, if it is in use.
• intelmq.bots.experts.ripe (PR#2461 by Mikk Margus Möll):
  • Handle "No abuse contact found for" messages for non-ASN resources

Outputs

• intelmq.bots.outputs.stomp.output (PR#2408 and PR#2414 by Jan Kaliszewski):
  • Drop support for versions of stomp.py older than 4.1.12.
  • Update the code to support new versions of stomp.py, including the latest (8.1.0).
  • Add support for authentication based on STOMP login and passcode, introducing three
    new configuration parameters (see above: Configuration).
  • Add support for loading the system’s default CA certificates, as an alternative to
    specifying the CA certificate(s) file path explicitly (see above: Configuration).
  • Fix (by carefully targeted monkey patching) certain security problems caused by
    SSL-related weaknesses that some versions of stomp.py suffer from.
  • Fix AttributeError caused by attempts to get unset attributes of StompOutputBot
    (ssl_ca_cert et consortes).
  • Add coercion of the port config parameter to int.
  • Add implementation of the check hook (verifying, in particular, accessibility
    of necessary file(s)).
  • Add stomp.py version check (raise MissingDependencyError if not >=4.1.12).
  • Minor fixes/improvements and some refactoring (see also above: Core...).
• intelmq.bots.outputs.stomp.output (PR#2423 by Kamil Mankowski):
  • Try to reconnect on NotConnectedException.
• intelmq.bots.outputs.smtp_batch.output (PR #2439 by Edvard Rejthar):
  • Fix ability to send with the default bcc
• intelmq.bots.outputs.amqp: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).

Documentation

• Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
• Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
• Update/fix/improve the stuff related to the STOMP bots and integration with the n6's
  Stream API (PR#2408 and PR#2414 by Jan Kaliszewski).
• Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
• Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
• Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).

Packaging

• Add pendulum to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner).
• debian/control: in Suggests field, replace python3-stomp.py (>= 4.1.9) with
  python3-stomp (>= 4.1.12), i.e., fix the package name by removing the .py
  suffix and bump the minimum version to 4.1.12 (PR#2414 by Jan Kaliszewski).

Tests

Tools

• intelmq_psql_initdb:
  • got support for providing custom harmonization file, generating view for storing raw fields separately, and adding IF NOT EXISTS/OR REPLACE clauses (PR#2404 by Kamil Mankowski).
  • got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).

Contrib

Known issues

3.2.1 Important Bugfixes

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

All Bots

Fixes an issue which prevented bots from stopping gracefully after reloading.
As logrotate reloads all bots regularly, this bug affects most IntelMQ installations.

Reverse DNS Expert

Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses.
If the bot retrieved the PTR for 192.0.43.7, it was cached for 192.0.43.0/24 and used for all IP addresses in this range, for example for 192.0.43.8.
IntelMQ version 3.2.1 fixes this issue.

The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used.

3.2.0 Feature release: Running IntelMQ bots as Python Library

IEP007: Running IntelMQ bots as Python Library is implemented.

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi.
Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.

Core

• intelmq.lib.utils:
  • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
• Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
• Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
• intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorný).
• intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorný).
• intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorný).
• intelmq.lib.harmonization:
  • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný).
  • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
• intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.

Development

• CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).

Bots

Collectors

• intelmq.bots.collector.rt:
  • restrict python-rt to be below version 3.0 due to introduced breaking changes,
  • added support for Subject NOT LIKE queries,
  • added support for multiple values in ticket subject queries.
• intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).

Parsers

• intelmq.bots.parsers.shadowserver._config:
  • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
• intelmq.bots.parsers.shadowserver._config:
  • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
  • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
  • Added 'Accessible-SIP' report. (PR#2348)
  • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
  • Removed duplicate mappings from the 'Spam-URL' report. (PR#2348)
• intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorný).

Experts

• intelmq.bots.experts.sieve:
  • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus Möll).
• intelmq.bots.experts.cymru_whois:
  • Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
  • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
• intelmq.bots.experts.sieve:
  • Added :before and :after keywords (PR#2374)

Outputs

• intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
• intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
• intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)

Documentation

• API: update API installation to be aligned with the rewritten API, and clarify some missing steps.

Tests

• New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)

Tools

• intelmqsetup:
  • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Mańkowski, fixes #2354)
• contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)
• intelmq_psql_initdb:
  • Added parameter -o to set the output file destination. (by Sebastian Kufner)
• intelmqctl:
  • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)

Known Issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
• Breaking changes in 'rt' library (#2367).
• Stomp collector failed (#2342).
• Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
• intelmq_psql_initdb does not work for SQLite (#2202).
• intelmqsetup: should install a default state file (#2175).
• Misp Expert - Crash if misp event already exist (#2170).
• Turris greylist has been updated (#2167).
• Spamhaus CERT parser uses wrong field (#2165).
• Custom headers ignored in HTTPCollectorBot (#2150).
• intelmqctl log: parsing syslog does not work (#2097).
• Bash completion scripts depend on old JSON-based configuration files (#2094).
• Bot configuration examples use JSON instead of YAML (#2066).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
• Corrupt dump files when interrupted during writing (#870).

3.1.0 Feature Release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

• Upgraded syntax to Python 3.6 (mostly Format-Strings) using pyuprade (PR#2136 by Sebastian Wagner).

Core

• intelmq.lib.upgrades:
  • Refactor upgrade functions global configuration handling removing the old-style defaults configuration (PR#2058 by Sebastian Wagner).
  • Pass version history as parameter to upgrade functions (PR#2058 by Sebastian Wagner).
• intelmq.lib.message:
  • Fix and pre-compile the regular expression for harmonization key names and also check keys in the extra. namespace (PR#2059 by Sebastian Wagner, fixes #1807).
• intelmq.lib.bot.SQLBot was replaced by an SQLMixin in intelmq.lib.mixins.SQLMixin. The Generic DB Lookup Expert bot and the SQLOutput bot were updated accordingly.
  • Added support for MSSQL (PR#2171 by Karl-Johan Karlsson).
  • Added optional reconnect delay parameter (PR#2171 by Karl-Johan Karlsson).
• Added an ExpertBot class - it should be used by all expert bots as a parent class
• Introduced a module for IntelMQ related datatypes intelmq.lib.datatypes which for now only contains an Enum listing the four bot types
• Added a bottype attribute to CollectorBot, ParserBot, ExpertBot, OutputBot
• Introduces a module for IntelMQ processmanagers. The processmanagers were up until now part of the intelmqct script.
  They now reside in intelmq.lib.processmanager which also contains an interface definition the processmanager implementations must adhere to.
  Both the processmanagers and the intelmqctl script were cleaned up a bit.
  The LogLevel and ReturnType Enums were added to intelmq.lib.datatypes.
• intelmq.lib.bot:
  • Enhance behaviour if an unconfigured bot is started (PR#2054 by Sebastian Wagner).
  • Fix line recovery and message dumping of the ParserBot (PR#2192 by Sebastian Wagner).
    • Previously the dumped message was always the last message of a report if the report contained multiple lines leading to data-loss.
  • Fix crashing at start in multithreaded bots (PR#2236 by DigitalTrustCenter).
  • Added default_fields parameter to ParserBot (PR#2293 by Filip Pokorný)
• intelmq.lib.pipeline:
  • Changed BRPOPLPUSH to BLMOVE, because BRPOPLPUSH has been marked as deprecated by redis in favor of BLMOVE (PR#2149 and PR#2240 by Sebastian Waldbauer and Sebastian Wagner, fixes #1827, #2233).
• intelmq.lib.utils:
  • Added wrapper resolve_dns for querying DNS, with the support for recommended methods from dnspython package in versions 1 and 2.
  • Moved line filtering inside RewindableFileHandle for easier handling and limiting number of temporary objects.
• intelmq.lib.harmonization:
  • Fixed DateTime handling of naive time strings (previously assumed local timezone, now assumes UTC) (PR#2279 by Filip Pokorný, fixes #2278)
  • Removes tzone argument from DateTime.from_timestamp and DateTime.from_epoch_millis
  • DateTime.from_timstamp now also allows string argument
• Removes pytz global dependency
• Removed support for Python 3.6, including removing conditional dependencies and updating syntax to use features from newest versions. (fixes #2272)

Development

• Removed Python 3.6 from CI.
• Enabled tests against Python 3.11.

Bots

• Set the parent class of all bots to the correct bot class

Collectors

• intelmq.bots.collectors.mail._lib:
  • Add support for unverified SSL/STARTTLS connections (PR#2055 by Sebastian Wagner).
  • Fix exception handling for aborted IMAP connections (PR#2187 by Sebastian Wagner).
• intelmq.bots.collectors.blueliv: Fix Blueliv collector requirements (PR#2161 by Gethvi).
• intelmq.bots.collectors.github_api._collector_github_api: Added personal access token support (PR#2145 by Sebastian Waldbauer, fixes #1549).
• intelmq.bots.collectors.file.collector_file: Added file lock support, no more race conditions (PR#2147 by Sebastian Waldbauer, fixes #2128)
• intelmq.bots.collectors.shadowserver.collector_reports_api.py: Added file_format option to download reports in CSV format for better performance (PR#2246 by elsif2)

Parsers

• intelmq.bots.parsers.alienvault.parser_otx: Save CVE data in extra.cve instead of extra.CVE due to the field name restriction on lower-case characters (PR#2059 by Sebastian Wagner).

• intelmq.bots.parsers.anubisnetworks.parser: Changed field name format from extra.communication.http.x_forwarded_for_#1 to extra.communication.http.x_forwarded_for_1 due to the field name restriction on alphanumeric characters (PR#2059 by Sebastian Wagner).

• intelmq.bots.parsers.dataplane.parser:

  • Add support for additional feeds (PR#2102 by Mikk Margus Möll).
    • DNS Recursion Desired
    • DNS Recursion Desired ANY
    • DNS Version
    • Protocol 41
    • SMTP Greet
    • SMTP Data
    • Telnet Login
    • VNC/RFB Login
  • Fix event object creation (PR#2298 by DigitalTrustCenter).

• Removed intelmq.bots.parsers.malc0de: this bot was marked as deprecated and removed from feed due to offline status (PR#2184 by Tamas Gutsohn, fixes #2178).

• intelmq.bots.parsers.microsoft.parser_ctip:

  • New parameter overwrite (PR#2112 by Sebastian Wagner, fixes #2022).
  • Fix handling of field Payload.domain if it contains the same IP address as Payload.serverIp (PR#2144 by Mikk Margus Möll and Sebastian Wagner).
  • Handle Payload field with non-base64-encoded JSON content and numbered dictionaries (PR#2193 by Sebastian Wagner)

• intelmq.bots.parsers.shodan.parser (PR#2117 by Mikk Margus Möll):

  • Instead of keeping track of extra.ftp.<something>.parameters, FTP parameters are collected together into extra.ftp.features as a list of said features, reducing field count.
  • Shodan field rsync.modules is collected.
  • Conversion functions can raise NoValueException with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given in hostnames, which would then be passed into source.reverse_dns and fail to validate as a FQDN.
  • Variable _common_keys is moved out of the class.
  • _dict_dict_to_obj_list is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.g vulns.CVE-2010-0001.cvss, CVE-2010-0002.cvss etc.
  • _get_first to get the first item from a list, with NoValueException raised on empty lists.
  • _get_first_hostname to handle the first valid FQDN from a list of hostnames for hostnames in the Shodan banner, if there is one, and gives NoValueException otherwise.
  • ssl.cert.serial and ssl.dhparams.generator, which may return both integers and strings, are converted to strings.
  • Changes to method apply_mapping, such as reducing needless loop iterations, removing a big try-except, and adding the NoValueException handling described above.
  • Stops falsy values (False, 0) besides None from being filtered out.

• intelmq.bots.parsers.shadowserver._config:

  • Added support for Accessible AMQP, Device Identification Report (IPv4 and IPv6) (PR#2134 by Mateo Durante).
  • Added file name mapping for SSL-POODLE-Vulnerable-Servers IPv6 (file name scan6_ssl_poodle) (PR#2134 by Mateo Durante).
  • Added Malware-URL, Sandbox-Connection, Sandbox-DNS, Accessible-AMQP, Open-AnonymouIs-MQTT, Accessible-QUIC, Accessible-SSH, SYNful-Knock, and Special (PR#2227 by elsif2)
  • Removed legacy reports Amplification-DDoS-Victim, CAIDA-IP-Spoofer, Darknet, Drone, Drone-Brute-Force, IPv6-Sinkhole-HTTP-Drone, Microsoft-Sinkhole, and Sinkhole-HTTP-Drone (PR#2227 by elsif2).
  • Users storing events in a database should be aware that field names and types have been updated (PR#2227 by elsif2).
  • Corrected "Accessible-AMQP" message_length type (int) and added "STUN" support (PR#2235 by elsif2).
  • Added amplification factor to UDP scan reports (PR#2238 by elsif2).
  • Added version and build_date to "Vulnerable-HTTP" report (PR#2238 by elsif2).
  • The following field types have been standardized across all Shadowserver reports (PR#2246 by elsif2):
    destination.fqdn (validate_fqdn)
    destination.url (convert_http_host_and_url)
    extra.browser_trusted (convert_bool)
    extra.duration (convert_int)
    extra.end_time (convert_date_utc)
    extra.freak_vulnerable (convert_bool)
    extra.ok (convert_bool)
    extra.password (validate_to_none)
    extra.ssl_poodle (convert_bool)
    extra.status (convert_int)
    extra.uptime (convert_int)
    extra.version (convert_to_none)
    source.network (validate_network)
  • The following report field names have changed to better represent their values:
    scan_rsync:extra.password renamed to extra.has_password
    scan_elasticsearch:status renamed to http_code
  • Added Accessible-HTTP-proxy and Open-HTTP-proxy (PR#2246 by elsif2).
  • Added http_agent to the Honeypot-DDoS report and added the DDoS-Participant report (PR#2303 by elsif2)
  • Added Accessible-SLP, IPv6 Accesssible-SLP, IPv6-DNS-Open-Resolvers, and IPv6-Open-LDAP-TCP reports (PR#2311 by elsif2)
  • Standardized response_length to response_size in Accessible-ICS and Open-MSSQL (PR#2311 by elsif2)

• intelmq.bots.parsers.cymru.parser_cap_program: The parser mapped the hostname into source.fqdn which is not allowed by the IntelMQ Data Format. Added a check (PR#2215 by Sebastian Waldbauer, fixes #2169)

• intelmq.bots.parsers.generic.parser_csv:

  • Use RewindableFileHandle to u...

3.1.0 Release candidate

What's Changed

• BUG: utils/load_configuration: fix loading JSON with tabs in #2010
• CTIP parser: Handle TLP value "unknown" in #2009
• FIX: s/self.parameters/self/ in Shodan collector by @monoidic in #2020
• BUG: provide filename when reading a non-archived file from mail by @kapalex in #2021
• Add new bot: cut string by delimeter, max length by @mariuskarotkis in #1967
• Core fixes by @monoidic in #2027
• DOC: Fix typo by @schacht-certat in #2029
• Make update script less verbose by default by @schacht-certat in #2033
• Remove malwardomains parser by @schacht-certat in #2026
• Add new bot: domain valid by @mariuskarotkis in #1966
• Support Shadowserver Vulnerable SMTP server feed, fix #1984 by @monoidic in #2037
• Shadowserver feed updates by @schacht-certat in #2036
• Remove postgresql output bot by @schacht-certat in #2045
• DOC: remove mention of intelmqctl list bot json output by @schacht-certat in #2030
• Remove workarounds for XMPP bots by @schacht-certat in #2048
• Use the YAML unsafe loader instead of the safe loader by @schacht-certat in #2041
• Make the harmonization attribute of the bot private by @schacht-certat in #2049
• ENH: imap collector: support unverified connections in #2055
• ENH: utils: list_bots: strip descriptions in #2056
• lib/harm: minor change to allowed classification.types list in #2057
• ENH: small refactor of upgrade functions in #2058
• Add Honeypot HTTP Scan feed to Shadowserver parser by @monoidic in #2047
• IDF: Fix and enforce field name restrictions in #2059
• JinjaExpert - modify message fields using jinja2 by @schacht-certat in #2040
• Implement a clear class hierarchy by @schacht-certat in #2042
• DOCS: Docker-compose commands updated by @waldbauer-certat in #2035
• Refactoring intelmqctl by @schacht-certat in #2067
• ENH: lib/bot: fix behavior for unconfigured bots in #2054
• BUG: fix v300_pipeline_file_removal by @schacht-certat in #2083
• Adapt ansible workflow to ansible tests by @schacht-certat in #2081
• BUG: use correct private parameter variable by @schacht-certat in #2087
• DOCS: fix some sphinx warnings by @schacht-certat in #2065
• ENH: make json output indented to make it better readable by humans by @schacht-certat in #2079
• ENH: use the upstream ansible tasks playbook by @schacht-certat in #2089
• Return an INFO string if log file does not yet exist by @schacht-certat in #2100
• MAINT: update the url for the vagrant+ansible test repository by @schacht-certat in #2061
• FIX: statistics_host returns if no host value is given by @waldbauer-certat in #2104
• DOCS: Updated docker installation instructions by @waldbauer-certat in #2069
• BUG: fix logrotate ownership options in #2111
• Add support for new feeds to Dataplane parser by @monoidic in #2102
• Add RPZ file output bot by @mariuskarotkis in #1962
• Add new bot: bro output file by @mariuskarotkis in #1963
• DOC: restructure installation documentation in #2113
• ENH: ctip parser: added parameter overwrite in #2112
• Add new bot: cut string from string by @mariuskarotkis in #1965
• ENH: intelmqsetup: revise installation of manager in #2114
• DOC: enhance misp integration document in #2115
• ENH: Shodan parser: handle invalid hostnames, move _common_keys out of class, adjust apply_mapping by @monoidic in #2117
• Add JSON parser function to Jinja2 templates by @creideiki in #2120
• DOC: feeds.yaml: fix HIBP nginx config by @sebix in #2126
• FIX: Fixes update-database script on the last few days of a month. by @gethvi in #2121
• DOC: add changelog for PR#2121 by @sebix in #2129
• Docs: Remove dead IRC channel by @sebix in #2130
• Added new shadowserver files to _config.py by @mateodurante in #2134
• DOC: Update changelog and docs for shadowserver PR#2134 by @sebix in #2135
• doc: feeds whishlist: add misp warning lists by @sebix in #2141
• docs: feeds whishlist: add avast ioc repo by @sebix in #2139
• ENH: upgrade Python syntax to 3.6+ by @sebix in #2136
• MS CTIP Azure feed invalid hostname filtering by @monoidic in #2144
• TST: also test on python 3.10, switch to pytest by @sebix in #2142
• FIX: CodeQL exponential backtracking on strings by @waldbauer-certat in #2148
• FIX: Added vscode to gitignore & automated docker build with ACT by @waldbauer-certat in #2153
• FIX: threshold expert: configure timeout using Redis cache mixin by @creideiki in #2155
• FIX: Fixes Blueliv collector requirements by @gethvi in #2161
• Add fix Blueliv collector requirements to CHANGELOG.md by @gutsohnCERT in #2173
• FIX: Moved SQLBot to SQLMixin namespace in generic-db-lookup by @waldbauer-certat in #2146
• DOC: installation: remove EOL Fedora versions by @sebix in #2186
• BUG: mail collectors: fix IMAP abort handling by @wagner-intevation in #2187
• bug: fix ctl process manager initialization non-interactive by @wagner-intevation in #2189
• doc: ms azure collector: more precise requirement name by @wagner-intevation in #2190
• intelmqsetup: fix typo in message by @sebix in #2181
• gitignore: add coverage file by @sebix in #2182
• ENH: new parameter autoupdate_cached_database by @sebix in #2180
• tst: fix reverse_dns test by @wagner-intevation in #2208
• pkg: add requests_mock as dev dependency by @sebix in #2210
• intelmqsetup: Fix #2197 by @wagner-intevation in #2198
• Fix reconnection in SQL Mixin by @wagner-intevation in #2201
• ParserBot: Fix line recovery and message dumping by @wagner-intevation in #2192
• FIX: Added check if cymru hostname is an ip address by @waldbauer-certat in #2215
• bug: fix ctip parser handling JSON Payload content by @wagner-intevation in #2193
• docs: workaround for sphinx-doc/sphinx#10701 by @wagner-intevation in #2225
• FIX: Remove AutoShun from IntelMQ as the feed is discontinued by @waldbauer-certat in #2214
• SQL: add MSSQL support and allow storing only a subset of event fields by @creideiki in #2172
• ENH: Added personal access token support to GitHub Collector by @waldbauer-certat in #2145
• FIX: Use redis BLMOVE instead of BRPOPLPUSH (deprecated) by @waldbauer-certat in #2149
• postgresql output: escape all null bytes by @wagner-intevation in #2223
• Remove malc0de feed due to offline by @gutsohnCERT in #2184
• TST: Fixed regex to hold gcc version by @waldbauer-certat in #2216
• Shadowserver 202207 by @elsif2 in https://github.com/certtools/intelmq/p...

3.0.2 Bugfix release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

• intelmq.lib.bot.CollectorBot: Fixed an issue with within the new_report function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR#2106 by Sebastian Waldbauer, fixes #2098).
• intelmq.lib.bot.Bot: Make private members __is_multithreadable and __collector_empty_process protected members _is_multithreadable and _collector_empty_process to make them easily modifiable by Bot classes (PR#2109 by Sebastian Wagner, fixes #2108).
  Also affected and adapted bots by this change are:
  • intelmq.bots.collectors.api.collector_api
  • intelmq.bots.collectors.stomp.collector
  • intelmq.bots.experts.splunk_saved_search.expert
  • intelmq.bots.experts.threshold.expert
  • intelmq.bots.outputs.file.output
  • intelmq.bots.outputs.misp.output_api
  • intelmq.bots.outputs.misp.output_feed
  • intelmq.bots.outputs.tcp.output
  • intelmq.bots.outputs.udp.output
• intelmq.lib.cache: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR#2104 by Sebastian Waldbauer, fixes #2103).

Bots

Experts

• intelmq.bots.experts.domain_suffix.expert: Only print skipped database update message if verbose mode is active (PR#2107 by Sebastian Wagner, fixes #2016).

Documentation

• Add configuration upgrade steps for 3.0 to NEWS (PR#2101 by Sebastian Wagner).

Known issues

See open bug reports for a more detailed list.

• ParserBot: erroneous raw line recovery in error handling (#1850).

3.0.1 Bugfix Release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

• intelmq.lib.bot_debugger: Fix accessing the bot's destination queues (PR#2027 by Mikk Margus Möll).
• intelmq.lib.pipeline: Fix handling of load_balance parameter (PR#2027 by Mikk Margus Möll).
• intelmq.lib.bot: Fix handling of parameter destination_queues if value is an empty dictionary (PR#2051 by Sebastian Wagner, fixes #2034).

Bots

Collectors

• intelmq.bots.collectors.shodan.collector_stream: Fix access to parameters, the bot wrongly used self.parameters (PR#2020 by Mikk Margus Möll).
• intelmq.bots.collectors.mail.collector_mail_attach: Add attachment file name as extra.file_name also if the attachment is not compressed (PR#2021 by Alex Kaplan).
• intelmq.bots.collectors.http.collector_http_stream: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.microsoft.parser_ctip: Map Payload.domain to destination.fqdn instead of extra.payload.domain as it matches to destination.ip from DestinationIp (PR#2023 by Sebastian Wagner).
• Removed intelmq.bots.parsers.malwaredomains because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
• intelmq.bots.parsers.shadowserver.config:
  • Add support for feed "Vulnerable SMTP Server" (PR#2037 by Mikk Margus Möll).
  • Fix differentiation between feeds "Accessible HTTP" and "Vulnerable HTTP" (PR#2037 by Mikk Margus Möll, fixes #1984).
  • Add support for the new feeds Microsoft Sinkhole Events Report, Microsoft Sinkhole HTTP Events Report (PR#2036 by Birger Schacht).
  • Complement feed mappings and documentation for feeds with IPv4 and IPv6 variants (PR#2046 by Mikk Margus Möll and Sebastian Wagner).
  • Feed names with and without the optional IPv4/IPv6 postfix can be used now consistently.
  • Add support for feed "Honeypot HTTP Scan" (PR#2047 by Mikk Margus Möll).
  • Update filename mapping for changed filename of feed "Accessible-MSRDPUDP" (PR#2060 by abr4xc).

Experts

• intelmq.bots.experts.gethostbyname.expert: Handle numeric values for the gaierrors_to_ignore parameter (PR#2073 by Sebastian Wagner, fixes #2072).
• intelmq.bots.experts.filter.expert: Fix handling of empty-string parameters not_after and not_before (PR#2075 by Sebastian Wagner, fixes #2074).

Outputs

• intelmq.bots.outputs.mcafee.output_esm_ip: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
• intelmq.bots.outputs.misp.output_api: Fix access to parameters, the bot wrongly used self.parameters (by Sebastian Wagner).
• intelmq.bots.outputs.smtp.output: Add Content-Disposition-header to the attachment, fixing the display in Mail Clients as actual attachment (PR#2052 by Sebastian Wagner, fixes #2018).

Documentation

• Various formatting fixes (by Sebastian Wagner).
• Removed the malwaredomains feed from the feeds list because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
• Update Docker installation instructions (PR#2035 by Sebastian Waldbauer).

Packaging

• intelmq-update-database crontab: Add missing recordedfuture_iprisk update call (by Sebastian Wagner).

Tests

• Replace calls to deprecated/undocumented logging.warn with logging.warning (by Sebastian Wagner, fixes #2013).
• intelmq.tests.bots.experts.rdap.test_expert: Declare cache use, fixes build failures (by Sebastian Wagner, fixes #2014).
• intelmq.tests.bots.collectors.mail.test_collector_attach: Test text attachment (by Sebastian Wagner).

Tools

• intelmqctl:
  • Also honour parameters from environment variables (PR#2068 by Sebastian Wagner, fixes #2063).
  • Fix management actions (start/stop/status/reload/restart) for groups (PR#2086 by Sebastian Wagner, fixes #2085).
  • Do not use hardcoded logging path in /opt/intelmq, use the internal default instead (PR#2092 by Sebastian Wagner, fixes #2091).

Known issues

See open bug reports for a more detailed list.

• ParserBot: erroneous raw line recovery in error handling (#1850).

3.0.0 Major release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Configuration

• The BOTS file is no longer used and has been removed (by Sebastian Wagner).
• The defaults.conf file is no longer used and has been removed (PR#1814 by Birger Schacht).
• The pipeline.conf file is no longer used and has been removed (PR#1849 by Birger Schacht).
• The runtime.conf was renamed to runtime.yaml and is now in YAML format (PR#1812 by Birger Schacht).

Core

• intelmq.lib.harmonization:
  • New class ClassificationTaxonomy with fixed list of taxonomies and sanitiation
• intelmq.lib.bot:
  • Handle InvalidValue exceptions upon message retrieval by dumping the message instead of repeating endlessly (#1765, PR#1766 by Filip Pokorný).
  • Rewrite of the parameter loading and handling, getting rid of the parameters member (PR#1729 by Birger Schacht).
  • The pipeline is now initialized before the call of init to allow bots accessing data directly on startup/initialization for cleanup or maintenance tasks (PR#1982 by Sebastian Wagner).
• intelmq.lib.exceptions:
  • InvalidValue: Add optional parameter object (PR#1766 by Filip Pokorný).
• intelmq.lib.utils:
  • New function list_all_bots to list all available/installed bots as replacement for the BOTS file (#368, #552, #644, #757, #1069, #1750, PR#1751 by Sebastian Waldbauer).
  • New function get_bots_settings to return the effective bot parameters, with global parameters applied.
  • Removed deprecated function create_request_session_from_bot (PR#1997 by Sebastian Wagner, #1404).
  • parse_relative: Add support for parsing minutes and seconds (PR#1857 by Sebastian Wagner).
• intelmq.lib.bot_debugger:
  • Set bot's logging_level directly in __init__ before the bot's initialization by changing the default value (by Sebastian Wagner).
  • Rewrite load_configuration_patch by adapting it to the parameter and configuration rewrite (by Sebastian Wagner).
  • Do not rely on the runtime configuration's group setting of bots to determine the required message type of messages given on the command line (PR#1949 by Sebastian Wagner).

Development

• rewrite_config_files.py: Removed obsolete BOTS-file-related rewriting functionality.
• A Github action that checks for reuse compliance of all the license and copyright headers was added.
• PyYAML is no longer a required dependency for development environments, all calls to it have been replaced by ruamel.yaml (by Sebastian Wagner).

Data Format

The IntelMQ Data Harmonization ("DHO") is renamed to IntelMQ Data Format ("IDF"). Internal files remain and work the same as before (PR#1818 by Sebastian Waldbauer, fixes 1810).
Update allowed classification fields to version 1.3 (2021-05-18) (fixes #1409, #1476).

• The taxonomy abusive content has been renamed to abusive-content.
• The taxonomy information content security has been renamed to information-content-security.
  • The validation of type unauthorised-information-access has been fixed, a bug prevented the use of it.
  • The validation of type unauthorised-information-modification has been fixed, a bug prevented the use of it.
  • The type leak has been renamed to data-leak.
  • The type dropzone has been removed. Taxonomy other with type other and identifier dropzone can be used instead. Ongoing discussion in the RSIT WG.
• The taxonomy intrusion attempts has been renamed to intrusion-attempts.
• For the taxonomy intrusions (PR#1993 by Sebastian Wagner, addresses #1409):
  • The type compromised has been renamed to system-compromise.
  • The type unauthorized-command has been merged into system-compromise.
  • The type unauthorized-login has been merged into system-compromise.
  • The type backdoor has been merged into system-compromise (PR#1995 by Sebastian Wagner, addresses #1409).
  • The type defacement has been merged into taxonomy information-content-security, type unauthorised-information-modification (PR#1994 by Sebastian Wagner, addresses #1409).
• The taxonomy information gathering has been rename to information-gathering.
• The taxonomy malicious code has been renamed to malicious-code.
  • The type c2server has been renamed to c2-server.
  • The type malware has been integrated into infected-system and malware-distribution, respectively (PR#1917 by Sebastian Wagner addresses #1409).
  • The type ransomware has been integrated into infected-system.
  • The type dga domain has been moved to the taxonomy other renamed dga-domain (PR#1992 by Sebastian Wagner fixes #1613).
• For the taxonomy 'availability', the type misconfiguration is new.
• For the taxonomy 'other', the type unknown has been renamed to undetermined.
• For the taxonomy 'vulnerable':
  • The type vulnerable client has been renamed to vulnerable-system.
  • The type vulnerable service has been renamed to vulnerable-system.

Bots

• The parameters handling of numerous bots has been refactored (PR#1751, PR#1729, by Birger Schacht, Sebastian Wagner, Sebastian Waldbauer).

Collectors

• Remove intelmq.bots.collectors.xmpp: one of the dependencies of the bot was deprecated and according to a short survey on the IntelMQ
  users mailinglist, the bot is not used by anyone. (https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html, PR#1761, closes #1614)
• intelmq.bots.collectors.mail._lib: Added parameter mail_starttls for STARTTLS in all mail collector bots (PR#1831 by Marius Karotkis, fixes #1128).
• Added intelmq.bots.collectors.fireeye: A bot that collects indicators from Fireeye MAS appliances (PR#1745 by Christopher Schappelwein).
• intelmq.bots.collectors.api.collector_api (PR#1987 by Mikk Margus Möll, fixes #1986):
  • Added UNIX socket capability.
  • Correctly close the IOLoop in the shutdown method to fix reload.
• intelmq.bots.collectors.rt.collector_rt (PR#1997 by Sebastian Wagner, #1404):
  • compatibility with the deprecated parameter unzip_attachment (removed in 2.1.0) was removed.

Parsers

• Added intelmq.bots.parsers.fireeye: A bot that parses hashes and URLs from Fireeye MAS indicators (PR#1745 by Christopher Schappelwein).
• intelmq.bots.parsers.shadowserver._config:
  • Improved the feed-mapping and all conversion functions (PR#1971 by Mikk Margus Möll).
• intelmq.bots.parsers.generic.parser_csv:
  • Fix handling of empty string values for parameter time_format (by Sebastian Wagner).

Experts

• intelmq.bots.experts.domain_suffix.expert:
  • Added --update-database option to update domain suffix database (by Sebastian Wagner).
  • Fix check method: load database with UTF-8 encoding explicitly (by Sebastian Wagner).
• Added intelmq.bots.experts.http.expert_status: A bot that fetches the HTTP Status for a given URI and adds it to the message (PR#1789 by Birger Schacht, fixes #1047 partly).
• Added intelmq.bots.experts.http.expert_content: A bot that fetches an HTTP resource and checks if it contains a specific string.
• Added intelmq.bots.experts.lookyloo.expert: A bot that sends requests to a lookyloo instance & adds screenshot_url to the event (PR#1844 by Sebastian Waldbauer, fixes #1048).
• Added intelmq.bots.experts.rdap.expert: A bot that checks the rdap protocol for an abuse contact for a given domain.
• intelmq.bots.experts.sieve.expert:
  • Add operators for comparing lists and sets (PR#1895 by Mikk Margus Möll):
    • :equals
    • :overlaps
    • :supersetof
    • :subsetof
    • :equals
  • Add support for comparing boolean values (PR#1895 by Mikk Margus Möll).
  • Add support for rule negation with ! (PR#1895, PR#1923 by Mikk Margus Möll).
  • Add support for values types float, int, bool and string for all lists items (PR#1895 by Mikk Margus Möll).
  • Add actions for lists (PR#1895 by Mikk Margus Möll).
    • append
    • append! (forced/overwriting)
  • Rewrite the rule-processing and operator-handling code to make it more comprehensible and extensible (PR#1895, PR#1923 by Mikk Margus Möll).
  • Nested if statements, plus mixed actions and actions in the same scope (PR #1923 by Mikk Margus Möll).
  • The attribute manipulation actions add, add! and update support non-string (bool/int/float) values (PR #1923 by Mikk Margus Möll).
  • Drop the :notcontains operator, as it made is redundant by generic negation: ! foo :contains 'x' instead of foo :notcontains 'x' (PR#1957 by Mikk Margus Möll).
  • Split string and numeric matches into single- and multivalued variants, with the relevant new operators :in, :containsany and :regexin for string lists, and :in for numeric value lists (PR#1957 by Mikk Margus Möll).
    • Removed the == operator for lists, with the previous meaning of :in. Have a look at the NEWS.md for more information.
• Added intelmq.bots.experts.uwhoisd: A bot that fetches the whois entry from a uwhois-instance (PR#1918 by Raphaël Vinot).
• Removed deprecated intelmq.bots.experts.ripencc_abuse_contact.expert. It was replaced by intelmq.bots.experts.ripe.expert and marked as deprecated in 2.0.0.beta1 (PR#1997 by Sebastian Wagner, #1404).
• intelmq.bots.experts.modify.expert:
  • Removed compatibility with deprecated configuration format before 1.0.0.dev7 (PR#1997 by Sebastian Wagner, #1404).
• Added intelmq.bots.experts.aggregate: A bot that aggregate events based upon given fields & a timespan. (PR#1959 by Sebastian Waldbauer)
• Added intelmq.bots.experts.tuency: A bot that queries the IntelMQ API of a tuency instance (PR#1857 by Sebastian Wagner, fixes #1856).

Outputs

• Remove intelmq.bots.outputs.xmpp: one of the dependencies of the bot was de...

3.0.0 Release candidate 1

This is just an intermediate unstable release towards 3.0.0. Please do not use it in production.

https://intelmq.readthedocs.io/

3.0.0 Beta 1

This is just an intermediate unstable release towards 3.0.0. Please do not use it in production.

https://intelmq.readthedocs.io/