3.1.0 Release candidate

What's Changed

• BUG: utils/load_configuration: fix loading JSON with tabs in #2010
• CTIP parser: Handle TLP value "unknown" in #2009
• FIX: s/self.parameters/self/ in Shodan collector by @monoidic in #2020
• BUG: provide filename when reading a non-archived file from mail by @kapalex in #2021
• Add new bot: cut string by delimeter, max length by @mariuskarotkis in #1967
• Core fixes by @monoidic in #2027
• DOC: Fix typo by @schacht-certat in #2029
• Make update script less verbose by default by @schacht-certat in #2033
• Remove malwardomains parser by @schacht-certat in #2026
• Add new bot: domain valid by @mariuskarotkis in #1966
• Support Shadowserver Vulnerable SMTP server feed, fix #1984 by @monoidic in #2037
• Shadowserver feed updates by @schacht-certat in #2036
• Remove postgresql output bot by @schacht-certat in #2045
• DOC: remove mention of intelmqctl list bot json output by @schacht-certat in #2030
• Remove workarounds for XMPP bots by @schacht-certat in #2048
• Use the YAML unsafe loader instead of the safe loader by @schacht-certat in #2041
• Make the harmonization attribute of the bot private by @schacht-certat in #2049
• ENH: imap collector: support unverified connections in #2055
• ENH: utils: list_bots: strip descriptions in #2056
• lib/harm: minor change to allowed classification.types list in #2057
• ENH: small refactor of upgrade functions in #2058
• Add Honeypot HTTP Scan feed to Shadowserver parser by @monoidic in #2047
• IDF: Fix and enforce field name restrictions in #2059
• JinjaExpert - modify message fields using jinja2 by @schacht-certat in #2040
• Implement a clear class hierarchy by @schacht-certat in #2042
• DOCS: Docker-compose commands updated by @waldbauer-certat in #2035
• Refactoring intelmqctl by @schacht-certat in #2067
• ENH: lib/bot: fix behavior for unconfigured bots in #2054
• BUG: fix v300_pipeline_file_removal by @schacht-certat in #2083
• Adapt ansible workflow to ansible tests by @schacht-certat in #2081
• BUG: use correct private parameter variable by @schacht-certat in #2087
• DOCS: fix some sphinx warnings by @schacht-certat in #2065
• ENH: make json output indented to make it better readable by humans by @schacht-certat in #2079
• ENH: use the upstream ansible tasks playbook by @schacht-certat in #2089
• Return an INFO string if log file does not yet exist by @schacht-certat in #2100
• MAINT: update the url for the vagrant+ansible test repository by @schacht-certat in #2061
• FIX: statistics_host returns if no host value is given by @waldbauer-certat in #2104
• DOCS: Updated docker installation instructions by @waldbauer-certat in #2069
• BUG: fix logrotate ownership options in #2111
• Add support for new feeds to Dataplane parser by @monoidic in #2102
• Add RPZ file output bot by @mariuskarotkis in #1962
• Add new bot: bro output file by @mariuskarotkis in #1963
• DOC: restructure installation documentation in #2113
• ENH: ctip parser: added parameter overwrite in #2112
• Add new bot: cut string from string by @mariuskarotkis in #1965
• ENH: intelmqsetup: revise installation of manager in #2114
• DOC: enhance misp integration document in #2115
• ENH: Shodan parser: handle invalid hostnames, move _common_keys out of class, adjust apply_mapping by @monoidic in #2117
• Add JSON parser function to Jinja2 templates by @creideiki in #2120
• DOC: feeds.yaml: fix HIBP nginx config by @sebix in #2126
• FIX: Fixes update-database script on the last few days of a month. by @gethvi in #2121
• DOC: add changelog for PR#2121 by @sebix in #2129
• Docs: Remove dead IRC channel by @sebix in #2130
• Added new shadowserver files to _config.py by @mateodurante in #2134
• DOC: Update changelog and docs for shadowserver PR#2134 by @sebix in #2135
• doc: feeds whishlist: add misp warning lists by @sebix in #2141
• docs: feeds whishlist: add avast ioc repo by @sebix in #2139
• ENH: upgrade Python syntax to 3.6+ by @sebix in #2136
• MS CTIP Azure feed invalid hostname filtering by @monoidic in #2144
• TST: also test on python 3.10, switch to pytest by @sebix in #2142
• FIX: CodeQL exponential backtracking on strings by @waldbauer-certat in #2148
• FIX: Added vscode to gitignore & automated docker build with ACT by @waldbauer-certat in #2153
• FIX: threshold expert: configure timeout using Redis cache mixin by @creideiki in #2155
• FIX: Fixes Blueliv collector requirements by @gethvi in #2161
• Add fix Blueliv collector requirements to CHANGELOG.md by @gutsohnCERT in #2173
• FIX: Moved SQLBot to SQLMixin namespace in generic-db-lookup by @waldbauer-certat in #2146
• DOC: installation: remove EOL Fedora versions by @sebix in #2186
• BUG: mail collectors: fix IMAP abort handling by @wagner-intevation in #2187
• bug: fix ctl process manager initialization non-interactive by @wagner-intevation in #2189
• doc: ms azure collector: more precise requirement name by @wagner-intevation in #2190
• intelmqsetup: fix typo in message by @sebix in #2181
• gitignore: add coverage file by @sebix in #2182
• ENH: new parameter autoupdate_cached_database by @sebix in #2180
• tst: fix reverse_dns test by @wagner-intevation in #2208
• pkg: add requests_mock as dev dependency by @sebix in #2210
• intelmqsetup: Fix #2197 by @wagner-intevation in #2198
• Fix reconnection in SQL Mixin by @wagner-intevation in #2201
• ParserBot: Fix line recovery and message dumping by @wagner-intevation in #2192
• FIX: Added check if cymru hostname is an ip address by @waldbauer-certat in #2215
• bug: fix ctip parser handling JSON Payload content by @wagner-intevation in #2193
• docs: workaround for sphinx-doc/sphinx#10701 by @wagner-intevation in #2225
• FIX: Remove AutoShun from IntelMQ as the feed is discontinued by @waldbauer-certat in #2214
• SQL: add MSSQL support and allow storing only a subset of event fields by @creideiki in #2172
• ENH: Added personal access token support to GitHub Collector by @waldbauer-certat in #2145
• FIX: Use redis BLMOVE instead of BRPOPLPUSH (deprecated) by @waldbauer-certat in #2149
• postgresql output: escape all null bytes by @wagner-intevation in #2223
• Remove malc0de feed due to offline by @gutsohnCERT in #2184
• TST: Fixed regex to hold gcc version by @waldbauer-certat in #2216
• Shadowserver 202207 by @elsif2 in #2227
• FIX: Possible race condition in file collector by @waldbauer-certat in #2147
• tst: fix threshold expert redis host by @wagner-intevation in #2209
• 3.1.0 Release candidate 1 by @sebix in #2230

New Contributors

• @mateodurante made their first contribution in #2134
• @gutsohnCERT made their first contribution in #2173
• @wagner-intevation made their first contribution in #2187
• @elsif2 made their first contribution in #2227

Full Changelog: 3.0.2...3.1.0-rc1