Skip to content

Releases: certtools/intelmq

2.3.3 Bugfix release

31 May 19:31
2.3.3
af0d162
Compare
Choose a tag to compare
2.3.3 Bugfix release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

  • intelmq.lib.upgrade:
    • Added v233_feodotracker_browse for Abuse.ch Feodotracker Browse parser configuration adaption (PR#1941 by Sebastian Wagner).

Bots

Parsers

  • intelmq.bots.parsers.microsoft.parser_ctip:
    • Add support for new field SourceIpInfo.SourceIpv4Int (PR#1940 by Sebastian Wagner).
    • Fix mapping of "ConnectionType" fields, this is not protocol.application. Now mapped to extra.*.connection_type (PR#1940 by Sebastian Wagner).
  • intelmq.bots.parsers.shadowserver._config:
    • Add support for the new feeds Honeypot-Amplification-DDoS-Events, Honeypot-Brute-Force-Events, Honeypot-Darknet, IP-Spoofer-Events, Sinkhole-Events, Sinkhole-HTTP-Events, Vulnerable-Exchange-Server, Sinkhole-Events-HTTP-Referer (PR#1950, PR#1952, PR#1953, PR#1954, PR#1970 by Birger Schacht and Sebastian Wagner, PR#1971 by Mikk Margus Möll).

Experts

  • intelmq.bots.experts.splunk_saved_search.expert:
    • fixed erroneous string formatting (PR#1960 by Karl-Johan Karlsson).

Outputs

  • intelmq.bots.outputs.smtp.output:
    • Handle empty "fieldnames" parameter by sending no attachment (PR#1932 by Sebastian Wagner).

Documentation

  • Feeds:
    • Fixed Abuse.ch Feodotracker Browse parser configuration (PR#1941 by Sebastian Wagner fixes #1938).

Tests

  • intelmq.bots.parsers.html_table:
    • Added testcase for Abuse.ch Feodotracker Browse (PR#1941 by Sebastian Wagner).

Tools

  • intelmqsetup:
    • Set ownershop of state file path and its parent directory (PR#1911 by Sebastian Wagner).

Known issues

  • ParserBot: erroneous raw line recovery in error handling (#1850).
Assets 2

2.3.2 Bugfix release

27 Apr 10:20
2.3.2
e02f435
Compare
Choose a tag to compare
2.3.2 Bugfix release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

  • intelmq.lib.harmonization:
    • TLP type: accept value "yellow" for TLP level AMBER.

Bots

Collectors

  • intelmq.bots.collectors.shadowserver.collector_reports_api:
    • Handle timeouts by logging the error and continuing to next report (PR#1852 by Marius Karotkis and Sebastian Wagner, fixes #1823).

Parsers

  • intelmq.bots.parsers.shadowserver.config:
    • Parse and harmonize field end_time as date in Feeds "Drone-Brute-Force" and "Amplification-DDoS-Victim" (PR#1833 by Mikk Margus Möll).
    • Add conversion function convert_date_utc which assumes UTC and sanitizes the data to datetime (by Sebastian Wagner, fixes #1848).
  • intelmq.bots.parsers.shadowserver.parser_json:
    • Use the overwrite parameter for optionally overwriting the "feed.name" field (by Sebastian Wagner).
  • intelmq.bots.parsers.microsoft.parser_ctip:
    • Handle fields timestamp, timestamp_utc, source_ip, source_port, destination_ip, destination_port, computer_name, bot_id, asn, geo in Payload of CTIP Azure format (PR#1841, PR#1851 and PR#1879 by Sebastian Wagner).
  • intelmq.bots.parsers.shodan.parser:
    • Added support for unique keys and verified vulns (PR#1835 by Mikk Margus Möll).
  • intelmq.bots.parsers.cymru.parser_cap_program:
    • Fix parsing in whitespace edge case in comments (PR#1870 by Alex Kaplan, fixes #1862).

Experts

  • intelmq.bots.experts.modify:
    • Add a new rule to the example configuration to change the type of malicious-code events to c2server if the malware name indicates c2 (PR#1854 by Sebastian Wagner).
  • intelmq.bots.experts.gethostbyname.expert:
    • Fix handling of parameter gaierrors_to_ignore with value None (PR#1890 by Sebastian Wagner, fixes #1886).

Outputs

  • intelmq.bots.outputs.elasticsearch: Fix log message on required elasticsearch library message (by Sebastian Wagner).

Documentation

  • dev/data-harmonization: Fix taxonomy name "information gathering" should be "information-gathering" (by Sebastian Wagner).

Tests

  • intelmq.tests.bots.parsers.microsoft.test_parser_ctip_azure:
    • Add test case for TLP level "YELLOW".

Known issues

  • ParserBot: erroneous raw line recovery in error handling (#1850).
Assets 2

2.3.1 Bugfix release

25 Mar 14:28
2.3.1
0ecf90b
Compare
Choose a tag to compare
2.3.1 Bugfix release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

Core

  • intelmq.lib.utils:
    • log: Handle null value for logging parameter logging_max_size (PR#1786 by Sebastian Wagner, fixes #1778).
  • intelmq.lib.pipeline:
    • Amqp._get_queues: Check virtual host when retrieving queue sizes. Fixes output of intelmqctl check for orphaned queues if AMQP is used and the AMQP user has access to more virtual hosts (PR#1830 by Sebastian Wagner, fixes #1746).

Bots

Collectors

  • intelmq.bots.collectors.shadowserver.collector_reports_api: Added debug logging to show number of downloaded reports and download size (PR#1826 by Sebastian Wagner, partly addresses #1688 and #1823).

Parsers

  • intelmq.bots.parsers.cymru.parser_cap_program:
    • Adapt parser to new upstream format for events of category "bruteforce" (PR#1795 by Sebastian Wagner, fixes 1794).
  • intelmq.bots.parsers.shodan.parser:
    • Support nested conversions, improved protocol detection and extended Shodan parser mappings (PR#1821 by Mikk Markus Möll).

Documentation

  • Add missing newlines at end of docs/_static/intelmq-manager/*.png.license files (PR#1785 by Sebastian Wagner, fixes #1777).
  • Ecosystem: Revise sections on intelmq-cb-mailgen and fody (PR#1792 by Bernhard Reiter).
  • intelmq-api: Add documentation about necessary write permission for the session database file (PR#1798 by Birger Schacht, fixes intelmq-api#23).
  • FAQ: Section on redis socket permissions: set only minimal necessary permissions (PR#1809 by Sebastian Wagner).
  • Add document on hardware requirements (PR#1811 by Sebastian Wagner).
  • Feeds: Added Shodan Country Stream (by Sebastian Wagner).

Tests

  • Add missing newlines at end of various test input files (PR#1785 by Sebastian Wagner, fixes #1777).
  • intelmq.tests.bots.parsers.shodan.test_parser: Add test cases for new code (PR#1821 by Mikk Markus Möll).
  • intelmq.tests.lib.test_harmonization.test_datetime_convert: Only run this test in timezone UTC (PR#1825 by Sebastian Wagner).

Tools

  • intelmqsetup:
    • Also cover required directory layout and file permissions for intelmq-api (PR#1787 by Sebastian Wagner, fixes #1783).
    • Also cover webserver and sudoers configuration for intelmq-api and intelmq-manger (PR#1805 by Sebastian Wagner, fixes #1803).
  • intelmqctl:
    • Do not log an error message if logging to file is explicitly disabled, e.g. in calls from intelmsetup. The error message would not be useful for the user and is not necessary.

Known issues

  • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
  • Corrupt dump files when interrupted during writing (#870).
  • CSV line recovery forces Windows line endings (#1597).
  • intelmqdump: Honor logging_path variable (#1605).
  • Timeout error in mail URL fetcher (#1621).
  • Shadowserver Parser: Drone feed has (also?) application protocol in type field (mapped to transport protocol) (#1763).
Assets 2

2.3.0 Feature release

04 Mar 10:13
2.3.0
1d9ef49
Compare
Choose a tag to compare
2.3.0 Feature release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.

Configuration

Core

  • intelmq.lib.bot:
    • ParserBot.recover_line_json_stream: Make line parameter optional, as it is not needed for this method (by Sebastian Wagner).
    • Bot.argparser: Added class method _create_argparser (returns argparse.ArgumentParser) for easy command line arguments parsing (PR#1586 by Filip Pokorný).
    • Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorný).
    • Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
    • Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner).
  • intelmq.lib.upgrades:
    • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559, by Sebastian Wagner).
  • intelmq.lib.exceptions:
    • PipelineError: Remove unused code to format exceptions (by Sebastian Wagner).
  • intelmq.lib.utils:
    • create_request_session_from_bot:
      • Changed bot argument to optional, uses defaults.conf as fallback, renamed to create_request_session. Name create_request_session_from_bot will be removed in version 3.0.0 (PR#1524 by Filip Pokorný).
      • Fixed setting of http_verify_cert from defaults configuration (PR#1758 by Birger Schacht).
    • log: Use RotatingFileHandler for allow log file rotation without external tools (PR#1637 by Vasek Bruzek).
  • intelmq.lib.harmonization:
    • The IPAddress type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner).
    • DateTime.parse_utc_isoformat: Add parameter return_datetime to return datetime object instead of string ISO format (by Sebastian Wagner).
    • DateTime.convert: Fix utc_isoformat format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner).
    • DateTime.from_timestamp: Ensure that time zone information (+00:00) is always present (by Sebastian Wagner).
    • DateTime.__parse now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner).
  • intelmq.lib.upgrades:
    • Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner).
    • Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht).

Development

  • intelmq.bin.intelmq_gen_docs:
    • Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht).
    • Merged into docs/autogen.py (PR#1622 by Birger Schacht).

Bots

Collectors

  • intelmq.bots.collectors.eset.collector: Added (PR#1554 by Mikk Margus Möll).
  • intelmq.bots.collectors.http.collector_http:
    • Added PGP signature check functionality (PR#1602 by sinus-x).
    • If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner).
  • intelmq.bots.collectors.kafka.collector: Added (PR#1654 by Birger Schacht, closes #1634).
  • intelmq.bots.collectors.xmpp.collector: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
  • intelmq.bots.collectors.shadowserver.collector_api:
    • Added (#1683, PR#1700 by Birger Schacht).
    • Change file names in the report to .json instead of the original and wrong .csv (PR#1769 by Sebastian Wagner).
  • intelmq.bots.collectors.mail: Add content of the email's Date header as extra.email_date to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner).
  • intelmq.bots.collectors.http.collector_http_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
  • intelmq.bots.collectors.shodan.collector_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
  • intelmq.bots.collectors.twitter.collector_twitter:
    • Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner).
    • Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR#1754 by Sebastian Wagner).

Parsers

  • intelmq.bots.parsers.eset.parser: Added (PR#1554 by Mikk Margus Möll).
    • Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll).
  • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559, by Sebastian Wagner).
  • intelmq.bots.parsers.cznic.parser_haas: Added (PR#1560 by Filip Pokorný and Edvard Rejthar).
  • intelmq.bots.parsers.cznic.parser_proki: Added (PR#1599 by sinus-x).
  • intelmq.bots.parsers.key_value.parser: Added (PR#1607 by Karl-Johan Karlsson).
  • intelmq.bots.parsers.generic.parser_csv: Added new parameter compose_fields (by Sebastian Wagner).
  • intelmq.bots.parsers.shadowserver.parser_json: Added (PR#1700 by Birger Schacht).
  • intelmq.bots.parsers.shadowserver.config:
    • Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer).
    • Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer).
    • Ignore value 0 for source.asn and destination.asn in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner).
  • intelmq.bots.parsers.abusech.parser_ip: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus).
  • intelmq.bots.parsers.malwaredomainlist: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).

Experts

  • intelmq.bots.experts.rfc1918.expert:
    • Add support for ASNs (PR#1557 by Mladen Markovic).
    • Speed improvements.
    • More output in debug logging mode (by Sebastian Wagner).
    • Checks parameter length on initialization and in check method (by Sebastian Wagner).
  • intelmq.bots.experts.gethostbyname.expert:
    • Added parameter fallback_to_url and set to True (PR#1586 by Edvard Rejthar).
    • Added parameter gaierrors_to_ignore to optionally ignore other gethostbyname errors (#1553).
    • Added parameter overwrite to optionally overwrite existing IP addresses (by Sebastian Wagner).
  • intelmq.bots.experts.asn_lookup.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • The script update-asn-data is now deprecated and will be removed in version 3.0.
  • intelmq.bots.experts.maxmind_geoip.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • Added license_key parameter (PR#1524 by Filip Pokorný).
    • The script update-geoip-data is now deprecated and will be removed in version 3.0.
  • intelmq.bots.experts.tor_nodes.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • The script update-tor-nodes is now deprecated and will be removed in version 3.0.
  • intelmq.bots.experts.recordedfuture_iprisk.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • Added api_token parameter (PR#1524 by Filip Pokorný).
    • The script update-rfiprisk-data is now deprecated and will be removed in version 3.0.
  • Added intelmq.bots.experts.threshold (PR#1608 by Karl-Johan Karlsson).
  • Added intelmq.bots.experts.splunk_saved_search.expert (PR#1666 by Karl-Johan Karlsson).
  • intelmq.bots.experts.sieve.expert:
    • Added possibility to give multiple queue names for the path directive (#1462, by Sebastian Wagner).
    • Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer).
    • Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer).
  • intelmq.bots.experts.maxmind_geoip.expert:
    • Fixed handing over of overwrite parameter to event.add (PR#1743 by Birger Schacht).

Outputs

  • intelmq.bots.outputs.rt: Added Request Tracker output bot (PR#1589 by Marius Urkis).
  • intelmq.bots.outputs.xmpp.output: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
  • intelmq.bots.outputs.smtp.output: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner).

Documentation

  • Feeds:
    • Add ESET URL and Domain feeds (by Sebastian Wagner).
    • Remove unavailable HPHosts Hosts file feed (#1559 by Sebastian Wagner).
    • Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar).
    • Added CZ.NIC Proki feed (PR#1599 by sinus-x).
    • Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný).
    • Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner).
    • Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner).
    • Fixed parsing of the public field in the generated feeds documentation (PR#1641 by Birger Schacht).
    • Change the rate_limit parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds).
    • Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorný and Sebastian Wagner).
    • Added Shadowserver Reports API (by Sebastian Wagner).
    • Change the rate_limit parameter for many feeds from 2 days to the default one day (by Sebastian Wagner).
    • Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
  • Bots:
    • Enhance...
Read more
Assets 2

2.3.0 Release candidate 1

19 Feb 17:01
2.3.0.rc1
45d1da4
Compare
Choose a tag to compare
2.3.0 Release candidate 1 Pre-release
Pre-release 
2.3.0.rc1

2.3.0 Release candidate 1
Assets 2

2.2.3 Bugfix release

23 Dec 14:57
2.2.3
d6ddbe7
Compare
Choose a tag to compare
2.2.3 Bugfix release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/UPGRADING.md

Documentation

  • Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht).

Harmonization

  • See NEWS.md for information on a fixed bug in the taxonomy expert.

Bots

Collectors

  • intelmq.bots.rt.collector_rt: Log the size of the downloaded file in bytes on debug logging level.

Parsers

  • intelmq.bots.parsers.cymru.parser_cap_program:
    • Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt).
    • Add support for field additional_asns in optional information column.
  • intelmq.bots.parsers.microsoft.parser_ctip:
    • Fix mapping of DestinationIpInfo.DestinationIpConnectionType field (contained a typo).
    • Explicitly ignore field DestinationIpInfo.DestinationIpv4Int as the data is already in another field.
  • intelmq.bots.parsers.generic.parser_csv:
    • Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh).
    • Data fields containing - are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer).

Experts

  • intelmq.bots.experts.taxonomy.expert: Map type scanner to information-gathering instead of information gathering. See NEWS file for more information.

Tests

  • Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython.

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2

2.2.2 Bugfix release

28 Oct 19:48
2.2.2
b6c650e
Compare
Choose a tag to compare
2.2.2 Bugfix release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md

Core

  • intelmq.lib.upgrades:
    • Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist".

Bots

Parsers

  • intelmq.bots.parsers.shadowserver:
    • Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
    • Added support for the feeds Accessible Radmin and CAIDA IP Spoofer (PR#1600 by sinus-x).
  • intelmq.bots.parsers.anubisnetworks.parser: Fix parsing error where dst.ip was not equal to comm.http.host.
  • intelmq/bots/parsers/danger_rulez/parser: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus).
  • `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23).
  • intelmq.bots.parsers.microsoft.parser_ctip:
    • Add support for DestinationIpInfo.* and Signatures.Sha256 fields, used by the ctip-c2 feed (PR#1623 by Mikk Margus Möll).
    • Use extra.payload.text for the feed's field Payload if the content cannot be decoded (PR#1610 by Giedrius Ramas).

Experts

  • intelmq.bots.experts.cymru_whois:
    • Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606).
    • The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606).

Documentation

  • README:
    • Add Core Infrastructure Initiative Best Practices Badge.
  • Bots:
    • Generic CSV Parser: Add note on escaping backslashes (#1579).
    • Remove section of non-existing "Copy Extra" Bot.
    • Explain taxonomy expert.
    • Add documentation on n6 parser.
    • Gethostbyname expert: Add documentation how errors are treated.
  • Feeds:
    • Fixed bot modules of Calidog CertStream feed.
    • Add information on Microsoft CTIP C2 feed.

Packaging

  • In Debian packages, intelmqctl check and intelmqctl upgrade-config are executed in the postinst step (#1551, PR#1624 by Birger Schacht).

Tests

  • intelmq.tests.lib.test_pipeline: Skip TestAmqp.test_acknowledge on Travis with Python 3.8.
  • intelmq.tests.bots.outputs.elasticsearch.test_output: Refresh index intelmq manually to fix random test failures (#1593, PR#1595 by Zach Stone).

Tools

  • intelmqctl check:
    • For disabled bots which do not have any pipeline connections, do not raise an error, but only warning.
    • Fix check on source/destination queues for bots as well the orphaned queues.

Contrib

  • Bash completion scripts: Check both /opt/intelmq/ as well as LSB-paths (/etc/intelmq/ and /var/log/intelmq/) for loading bot information (#1561, PR#1628 by Birger Schacht).

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2

Bugfix release

30 Jul 13:45
2.2.1
4ecb114
Compare
Choose a tag to compare
Bugfix release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md

Core

  • intelmq.lib.upgrades:
    • Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
    • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559).
    • intelmq.lib.harmonization:
      • For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).

Development

  • Ignore line length (E501) in code-style checks altogether.

Bots

Collectors

  • intelmq.bots.collectors.misp: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)
  • intelmq.bots.collectors.stomp: Remove empty client.pem file.

Parsers

  • intelmq.bots.parsers.shadowserver.config:
    • Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
    • Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321).
  • intelmq.bots.parser.anubisnetworks.parser: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all.
  • intelmq.bots.parsers.generic.parser_csv: Allow values of type dictionary for parameter type_translation.
  • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559).
  • intelmq.bots.parsers.cymru.parser_cap_program: Add support for comment "username" for "scanner" category.
  • intelmq.bots.parsers.malwareurl.parser: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).

Experts

  • intelmq.bots.experts.maxmind_geoip: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.

Outputs

  • intelmq.bot.outputs.udp: Fix error handling on sending, had a bug itself.

Documentation

  • Feeds:
    • Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
  • Bots:
    • Overhaul of all bots' description fields (#1570).
  • User-Guide:
    • Overhaul pipeline configuration section and explain named queues better (#1577).

Tests

  • intelmq.tests.bots.experts.cymru: Adapt test_empty_result, remove test_unicode_as_name and test_country_question_mark (#1576).

Tools

  • intelmq.bin.intelmq_gen_docs: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form.
  • intelmq.bin.intelmqctl:
    • debug: In JSON mode, use dictionaries instead of lists.
    • debug: Add PATH to the paths shown.
    • check: Show $PATH environment variable if executable cannot be found.

Contrib

  • malware_name_mapping: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
  • Bash completion scripts search in wrong directory in packages (#1561).
  • Cymru Expert: Wrong Cache-Key Calculation (#1592).
Assets 2

Feature Release

18 Jun 08:26
2.2.0
91a0d40
Compare
Choose a tag to compare
Feature Release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md

Dropped support for Python 3.4.

Core

  • __init__: Changes to the path-handling, see User Guide, section /opt and LSB paths for more information
    • The environment variable INTELMQ_ROOT_DIR can be used to set custom root directories instead of /opt/intelmq/ (#805) in case of non LSB-path installations.
    • The environment variable ROOT_DIR can be used to set custom root directories instead of / (#805) in case of LSB-path installations.
  • intelmq.lib.exceptions: Added MissingDependencyError for show error messages about a missing library and how to install it (#1471).
    • Added optional parameter installed to show the installed version.
    • Added optional parameter additional_text to show arbitrary text.
  • Adding more type annotations for core libraries.
  • intelmq.lib.pipeline.Pythonlist.sleep: Drop deprecated method.
  • intelmq.lib.utils: write_configuration: Append a newline at end of configuration/file to allow proper comparisons & diffs.
  • intelmq.lib.test: BotTestCase drops privileges upon initialization (#1489).
  • intelmq.lib.bot:
    • New class OutputBot:
      • Method export_event to format/export events according to the parameters given by the user.
    • ParserBot: New methods parse_json_stream and recover_line_json_stream.
    • ParserBot.recover_line_json: Fix format by adding a list around the line data.
    • Bot.send_message: In debugging log level, the path to which the message is sent is now logged too.

Bots

  • Bots with dependencies: Use of intelmq.lib.exceptions.MissingDependencyError.

Collectors

  • intelmq.bots.collectors.misp.collector: Deprecate parameter misp_verify in favor of generic parameter http_verify_cert.
  • intelmq.bots.collectors.tcp.collector: Drop compatibility with Python 3.4.
  • intelmq.bots.collectors.stomp.collector:
    • Check the stomp.py version and show an error message if it does not match.
    • For stomp.py versions >= 5.0.0 redirect the stomp.PrintingListener output to debug logging.
  • intelmq.bots.collectors.microsoft.collector_azure: Support current Python library azure-storage-blob>= 12.0.0, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
  • intelmq.bots.collectors.amqp.collector_amqp: Require pika minimum version 1.0.
  • intelmq.bots.collectors.github_api.collector_github_contents_api: Added (PR#1481).

Parsers

  • intelmq.bots.parsers.autoshun.parser: Drop compatibility with Python 3.4.
  • intelmq.bots.parsers.html_table.parser: Drop compatibility with Python 3.4.
  • intelmq.bots.parsers.shadowserver.parser: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
  • intelmq.bots.parsers.taichung.parser:
    • Migrate to ParserBot.
    • Also parse geolocation information if available.
  • intelmq.bots.parsers.cymru.parser_full_bogons:
    • Migrate to ParserBot.
    • Add last updated information in raw.
  • intelmq.bots.parsers.anubisnetworks.parser: Add new parameter use_malware_familiy_as_classification_identifier.
  • intelmq.bots.parsers.microsoft.parser_ctip: Compatibility for new CTIP data format used provided by the Azure interface.
  • intelmq.bots.parsers.cymru.parser_cap_program: Support for openresolver type.
  • intelmq.bots.parsers.github_feed.parser: Added (PR#1481).
  • intelmq.bots.parsers.urlvir.parser: Removed, as the feed is discontinued (#1537).

Experts

  • intelmq.bots.experts.csv_converter: Added as converter to CSV.
  • intelmq.bots.experts.misp: Added (PR#1475).
  • intelmq.bots.experts.modify: New parameter maximum_matches.

Outputs

  • intelmq.bots.outputs.amqptopic:
    • Use OutputBot and export_event.
    • Allow formatting the routing key with event data by the new parameter format_routing_key (boolean).
  • intelmq.bots.outputs.file: Use OutputBot and export_event.
  • intelmq.bots.outputs.files: Use OutputBot and export_event.
  • intelmq.bots.outputs.misp.output_feed: Added, creates a MISP Feed (PR#1473).
  • intelmq.bots.outputs.misp.output_api: Added, pushes to MISP via the API (PR#1506, PR#1536).
  • intelmq.bots.outputs.elasticsearch.output: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513).

Documentation

  • Document usage of the INTELMQ_ROOT_DIR environment variable.
  • Added document on MISP integration possibilities.
  • Feeds:
    • Added "Full Bogons IPv6" feed.
    • Remove discontinued URLVir Feeds (#1537).

Packaging

  • setup.py do not try to install any data to /opt/intelmq/ as the behavior is inconsistent on various systems and with intelmqsetup we have a tool to create the structure and files anyway.
  • debian/rules:
    • Provide a blank state file in the package.
  • Patches:
    • Updated fix-intelmq-paths.patch.

Tests

  • Travis: Use intelmqsetup here too.
    • Install required build dependencies for the Debian package build test.
    • This version is no longer automatically tested on Python < 3.5.
    • Also run the tests on Python 3.8.
    • Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
  • Added tests for the new bot intelmq.bots.outputs.misp.output_feed (#1473).
  • Added tests for the new bot intelmq.bots.experts.misp.expert (#1473).
  • Added tests for intelmq.lib.exceptions.
  • Added tests for intelmq.lib.bot.OutputBot and intelmq.lib.bot.OutputBot.export_event.
  • Added IPv6 tests for intelmq.bots.parsers.cymru.parser_full_bogons.
  • Added tests for intelmq.lib.bot.ParserBot's new methods parse_json_stream and recover_line_json_stream.
  • intelmq.tests.test_conf: Set encoding to UTF-8 for reading the feeds.yaml file.

Tools

  • intelmqctl:
    • upgrade-config:
      • Allow setting the state file location with the --state-file parameter.
      • Do not require a second run anymore, if the state file is newly created (#1491).
      • New parameter no_backup/--no-backup to skip creation of .bak files for state and configuration files.
    • Only require psutil for the IntelMQProcessManager, not for process manager independent calls like upgrade-config or check.
    • Add new command debug to output some information for debugging. Currently implemented:
      • paths
      • environment variables
    • IntelMQController: New argument --no-file-logging to disable logging to file.
    • If dropping privileges does not work, intelmqctl will now abort (#1489).
  • intelmqsetup:
    • Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
    • Call intelmqctl upgrade-config and add argument for the state file path (#1491).
  • intelmq_generate_misp_objects_templates.py: Tool to create a MISP object template (#1470).
  • intelmqdump: New parameter -t or --truncate to optionally give the maximum length of raw data to show, 0 for no truncating.

Contrib

  • Added development-tools.
  • ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513).
  • Malware Name Mapping Downloader:
    • New parameter --mwnmp-ignore-adware.
    • The parameter --add-default supports an optional parameter to define the default value.

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2

Bugfix release

26 May 10:58
2.1.3
32fb175
Compare
Choose a tag to compare
Bugfix release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md

Requirements

  • The python library requests is (again) listed as dependency of the core (#1519).

Core

  • intelmq.lib.upgrades:
    • Harmonization upgrade: Also check and update regular expressions.
    • Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
    • Add function to migrate changed Taichung URL feed.
    • Check for discontinued Abuse.CH Zeus Tracker feed.
  • intelmq.lib.bot:
    • ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
    • start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
    • start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
    • _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
  • intelmq.lib.test:
    • BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
    • Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
    • Fix test for (allowed) errors and warnings.
  • intelmq.lib.exceptions:
    • InvalidKey: Add KeyError as parent class.
    • DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
  • intelmq.lib.pipeline:
    • Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
  • intelmq.lib.utils:
    • decode: Raise new DecodingError if decoding fails.

Harmonization

  • protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).

Bots

Collectors

  • intelmq.bots.collectors.mail.collector_mail_attach:
    • Fix handling of deprecated parameter name attach_unzip.
    • Fix handling of attachments without filenames (#1538).
  • intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
  • intelmq.bots.collectors.microsoft:
    • Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
    • intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.

Parsers

  • intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
  • intelmq.bots.parsers.taichung.parser: Support more types/classifications:
    • Application Compromise: Apache vulnerability & SQL injections
    • Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
    • C2 Sever: Attack controller
    • DDoS
    • DoS: DNS, DoS, Excess connection
    • IDS Alert / known vulnerability exploitation: backdoor
    • Malware: Malware Proxy
    • Warn on new unknown types.
  • intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
  • intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
  • intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
  • intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
  • intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
  • intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
  • intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
  • intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
  • intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
  • intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.

Experts

  • intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.

Outputs

  • intelmq.bots.outputs.mongodb:
    • Set default port 27017.
    • Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).

Documentation

  • Feeds:
    • Remove unavailable feed Abuse.CH Zeus Tracker.
    • Remove the field status, offline feeds should be removed.
    • Add a new field public to differentiate between private and public feeds.
    • Adding documentation URLs to nearly all feeds.
    • Remove unavailable Bitcash.cz feed.
    • Remove unavailable Fraunhofer DDos Attack feeds.
    • Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
    • Update information on Bambenek Feeds, many require a license now (#1525).
    • Remove discontinued Nothink Honeypot Feeds (#1537).
  • Developers Guide: Fix the instructions for /opt/intelmq file permissions.

Packaging

  • Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
  • Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
  • Add runtime dependency debianutils for the program which, which is required for intelmqctl.

Tests

  • Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
  • intelmq.tests.bots.experts.cymru_whois:
    • Drop missing ASN test, does not work anymore.
    • IPv6 to IPv4 test: Test for two possible results.
  • intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
  • intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
  • Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
  • Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
  • Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
  • Travis: Explicitly enable RabbitMQ management plugin.
  • intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).

Tools

  • intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
  • intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
  • intelmqctl:
    • IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
    • IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
  • intelmqdump:
    • Handle base64-type messages for show, editor and recovery actions.

Contrib

  • intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, hadiasghari/pyasn#62).

Known issues

  • HTTP stream collector: retry on regular connection problems? (#1435).
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Reverse DNS: Only first record is used (#877).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2