Updating to mailcow 2024-01e with Docker 25.0.3 breaks iptables / UFW usage on Debian 10

[codiflow]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
• ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
• ... I have understood that answers are voluntary and community-driven, and not commercial support.
• ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

Updating my machine like usual to the newest mailcow 2024-01e (and I think there was also a docker update) broke my UFW / iptables setup on one of my servers which is using Debian 10.

If I shutdown the mailcow container and reboot the machine everything is fine and I get an output from iptables -L and also from ufw status

I have two other docker containers on the same machine and they are working fine. I tried several combinations like shutting down all containers, reboot, check ufw/iptables and starting only other containers and not mailcow, reboot, check ufw/iptables.

The result was clear:

As soon as I start the mailcow docker containers with docker compose up both outputs break and also the firewall functionality. I can only bring it back by shutting down mailcow, reenabling UFW with ufw enable, restarting the machine and reenabling UFW again with ufw enable.

Maybe the issue has to do with the Netfilter Enhancements like stated here?

Logs:

docker compose logs -t -f

https://paste.armbian.com/izeqepuzim.yaml

(text was too long)

Steps to reproduce:

Shutdown mailcow with docker compose down
Reboot machine
Check ufw status

Output:

Status: active

To                         Action      From
--                         ------      ----
Nginx Full                 ALLOW       Anywhere                  
25/tcp                     ALLOW       Anywhere                  
465/tcp                    ALLOW       Anywhere                  
143/tcp                    ALLOW       Anywhere                  
993/tcp                    ALLOW       Anywhere                  
110/tcp                    ALLOW       Anywhere                  
995/tcp                    ALLOW       Anywhere                  
587/tcp                    ALLOW       Anywhere                  
4190/tcp                   ALLOW       Anywhere                  
22/tcp                     ALLOW       Anywhere                  
Nginx Full (v6)            ALLOW       Anywhere (v6)             
25/tcp (v6)                ALLOW       Anywhere (v6)             
465/tcp (v6)               ALLOW       Anywhere (v6)             
143/tcp (v6)               ALLOW       Anywhere (v6)             
993/tcp (v6)               ALLOW       Anywhere (v6)             
110/tcp (v6)               ALLOW       Anywhere (v6)             
995/tcp (v6)               ALLOW       Anywhere (v6)             
587/tcp (v6)               ALLOW       Anywhere (v6)             
4190/tcp (v6)              ALLOW       Anywhere (v6)             
22/tcp (v6)                ALLOW       Anywhere (v6)

iptables -L

Output:
The usual IP tables entries

Now start mailcow with docker compose up -d

Check ufw status

Output:

Status: inactive

Start ufw enable

Output:

Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
ERROR: Could not load logging rules

Check ufw reload

Output:

ERROR: Could not load logging rules

Check iptables -L

Output:

iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

Shutdown mailcow with docker compose down

Check commands again like above: Same result

Only way to bring everything up again is to either disable ufw with ufw disable and reboot the machine or to shutdown mailcow, reenabling UFW with ufw enable, restarting the machine and reenabling UFW again with ufw enable.

But I would really like to use both like I did for years now 😅

Which branch are you using?

master

Which architecture are you using?

x86

Operating System:

Debian GNU/Linux 10 (buster)

Server/VM specifications:

32GB RAM, 6 Cores

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

KVM

Docker version:

25.0.3

docker-compose version or docker compose version:

v2.24.5

mailcow version:

2024-01e

Reverse proxy:

nginx

Logs of git diff:

diff --git a/data/conf/postfix/main.cf b/data/conf/postfix/main.cf
index 572300db..8bd33a4c 100644
--- a/data/conf/postfix/main.cf
+++ b/data/conf/postfix/main.cf
@@ -110,7 +110,8 @@ smtpd_tls_auth_only = yes
 smtpd_tls_dh1024_param_file = /etc/ssl/mail/dhparams.pem
 smtpd_tls_eecdh_grade = auto
 smtpd_tls_exclude_ciphers = ECDHE-RSA-RC4-SHA, RC4, aNULL, DES-CBC3-SHA, ECDHE-RSA-DES-CBC3-SHA, EDH-RSA-DES-CBC3-SHA
-smtpd_tls_loglevel = 1
+# Edit 20221001 Changed from 1 to 2
+smtpd_tls_loglevel = 2
 
 # Mandatory protocols and ciphers are used when a connections is enforced to use TLS
 # Does _not_ apply to enforced incoming TLS settings per mailbox
@@ -173,3 +174,36 @@ parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks
 
 # DO NOT EDIT ANYTHING BELOW #
 # Overrides #
+
+postscreen_dnsbl_sites = wl.mailspike.net=127.0.0.[18;19;20]*-2
+  hostkarma.junkemailfilter.com=127.0.0.1*-2
+  list.dnswl.org=127.0.[0..255].0*-2
+  list.dnswl.org=127.0.[0..255].1*-4
+  list.dnswl.org=127.0.[0..255].2*-6
+  list.dnswl.org=127.0.[0..255].3*-8
+  ix.dnsbl.manitu.net*2
+  bl.spamcop.net*2
+  bl.suomispam.net*2
+  hostkarma.junkemailfilter.com=127.0.0.2*3
+  hostkarma.junkemailfilter.com=127.0.0.4*2
+  hostkarma.junkemailfilter.com=127.0.1.2*1
+  backscatter.spameatingmonkey.net*2
+  bl.ipv6.spameatingmonkey.net*2
+  bl.spameatingmonkey.net*2
+  b.barracudacentral.org=127.0.0.2*7
+  bl.mailspike.net=127.0.0.2*5
+  bl.mailspike.net=127.0.0.[10;11;12]*4
+  dnsbl.sorbs.net=127.0.0.10*8
+  dnsbl.sorbs.net=127.0.0.5*6
+  dnsbl.sorbs.net=127.0.0.7*3
+  dnsbl.sorbs.net=127.0.0.8*2
+  dnsbl.sorbs.net=127.0.0.6*2
+  dnsbl.sorbs.net=127.0.0.9*2
+  zen.spamhaus.org=127.0.0.[10;11]*8
+  zen.spamhaus.org=127.0.0.[4..7]*6
+  zen.spamhaus.org=127.0.0.3*4
+  zen.spamhaus.org=127.0.0.2*3
+
+# User Overrides
+myhostname = <REDACTED-DOMAIN>
+
diff --git a/data/conf/rspamd/local.d/history_redis.conf b/data/conf/rspamd/local.d/history_redis.conf
index 68a59b0c..77e1ae3d 100644
--- a/data/conf/rspamd/local.d/history_redis.conf
+++ b/data/conf/rspamd/local.d/history_redis.conf
@@ -1 +1 @@
-nrows = 1000;
+nrows = 10000;
diff --git a/docker-compose.yml b/docker-compose.yml
index df545c15..4157b033 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -609,36 +609,6 @@ services:

Logs of iptables -L -vn:

not working if mailcow was started, only output:
iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

Logs of ip6tables -L -vn:

not working if mailcow was started, only output:
iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

Logs of iptables -L -vn -t nat:

not working if mailcow was started, only output:
iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

Logs of ip6tables -L -vn -t nat:

not working if mailcow was started, only output:
iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.

DNS check:

DNS has been tested and it was working

[codiflow]

Maybe thats helpful too:

These updates have been installed today alongside with mailcow:

Start-Date: 2024-02-10  14:14:30
Commandline: /usr/bin/apt-get upgrade -y
Upgrade: containerd.io:amd64 (1.6.27-1, 1.6.28-1), libisccfg163:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libirs161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), bind9-host:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), dnsutils:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), sudo:amd64 (1.8.27-1+deb10u5, 1.8.27-1+deb10u6), libisc-export1100:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libisc1100:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), man-db:amd64 (2.8.5-2, 2.8.5-2+deb10u1), liblwres161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libdns-export1104:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libisccc161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libbind9-161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libdns1104:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), docker-compose-plugin:amd64 (2.24.2-1~debian.10~buster, 2.24.5-1~debian.10~buster), docker-ce:amd64 (5:25.0.1-1~debian.10~buster, 5:25.0.2-1~debian.10~buster), docker-ce-cli:amd64 (5:25.0.1-1~debian.10~buster, 5:25.0.2-1~debian.10~buster)
End-Date: 2024-02-10  14:15:08

Start-Date: 2024-02-10  14:42:57
Commandline: apt upgrade
Install: debsuryorg-archive-keyring:amd64 (2024.02.05+0~20240205.1+debian10~1.gbp343037, automatic)
Upgrade: php-common:amd64 (2:94+0~20240121.50+debian10~1.gbpe06825, 2:94+0~20240205.51+debian10~1.gbp6faa2e), docker-ce:amd64 (5:25.0.2-1~debian.10~buster, 5:25.0.3-1~debian.10~buster), docker-ce-cli:amd64 (5:25.0.2-1~debian.10~buster, 5:25.0.3-1~debian.10~buster)
End-Date: 2024-02-10  14:43:17

[erichk4]

Had a similar problem...

What i did:

• stop the mailcow
• stop docker service
• install nftables (apt install nftables), this installs the nft tool
• nft -f /etc/nftables.conf
• start docker
• start mailcow

[codiflow]

Tried your approach step by step – unfortunately this did not change anything for me. The issue persists. 😥

[erichk4]

Maybe your problem is ufw, i don't use it in my setup...

[codiflow]

I don't think so as UFW uses its own chains.

But I could find some other hint while playing around with docker daemon and mailcow. After shutting down mailcow, reloading UFW and restarting the docker daemon the docker daemon was not coming up again.

I found this on the log:
dockerd[3154]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create FILTER chain DOCKER: iptables failed: iptables -t filter -N DOCKER: iptables v1.8.2 (nf_tables): Chain already exists

I could "fix" it temporarily by setting DISABLE_NETFILTER_ISOLATION_RULE to y in .env and restarting the machine / docker daemon and the firewall. Now UFW and iptables work like before. So the cause most likely is within recent netfilter changes like this one: 087481a#diff-1765fe97b183010a3dae8d7c2051c1e1ec96926170cdf72e980c893e37027072R218

[Fighter456]

I've had a similar case. My mailcow currently is on 2024-01d but I noticed the outstanding update of docker to 25.0.3. I've performed the upgrade to the new docker version with apt-get dist-upgrade and then had the problem, that the docker engine could not start, resulting in the following

Do you want to continue? [Y/n] y
Reading changelogs... Done
(Reading database ... 37766 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce-cli (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Preparing to unpack .../docker-ce_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Preparing to unpack .../docker-ce-rootless-extras_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Setting up docker-ce-cli (5:25.0.3-1~debian.10~buster) ...
Setting up docker-ce-rootless-extras (5:25.0.3-1~debian.10~buster) ...
Setting up docker-ce (5:25.0.3-1~debian.10~buster) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

After some research in the logs I found out the following:

Feb 11 13:08:23 systemd[1]: Starting Docker Application Container Engine...
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.354218510+01:00" level=info msg="Starting up"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.354365090+01:00" level=warning msg="Running experimental build"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.355628755+01:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.449964795+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.473845925+01:00" level=info msg="Loading containers: start."
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.556482277+01:00" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.`" error="exit status 1"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.630432993+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Feb 11 13:08:23 dockerd[9790]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create FILTER chain DOCKER: iptables failed: iptables -t filter -N DOCKER: iptables v1.8.2 (nf_tables): Chain already exists
Feb 11 13:08:23 dockerd[9790]:  (exit status 1)

After verifying the rule DOCKER already exists I've removed it (iptables -D …) and mailcow afterwards starts without any problem.

I think that moby/moby#47303 has something to do with it. That's the pull request regarding to the change log entry Fix a bug where containers are unable to communicate over an internal network. in the docker engine version 25.0.3.

The error message from the logs come from here
https://github.com/akerouanton/docker/blob/990e95dcf08af03ada217eac6109c09063a935a5/libnetwork/drivers/bridge/setup_ip_tables_linux.go#L59 and is part of the initialization progress of docker.

Upgrade of mailcow to 2024-01e without any problem.

[FreddleSpl0it]

If you encounter Problems, please try to set DISABLE_NETFILTER_ISOLATION_RULE=y in mailcow.conf and do a docker compose up -d. But beware that this might open a Vulnerability. To be on the safe side, restrict the access to the mailcow docker network with something like:

iptables:
iptables -I DOCKER-USER ! -i br-mailcow -o br-mailcow -p tcp -m multiport --dport 3306,6379,8983,12345 -j DROP

nftables:
nft insert rule ip "filter" "DOCKER-USER" iifname != "br-mailcow" oifname "br-mailcow" tcp dport {3306, 6379, 8983, 12345} counter packets 0 bytes 0 drop

[codiflow]

Thanks for clarifying this.

So does this mean there will be no "fix" for this issue (is it an issue within mailcow then?) and the only solution to this problem is to disable the netfilter isolation rule?

[MaxXor]

I'm experiencing the same/similar (?) issue, but don't have ufw installed, only nftables. netfilter container is also restarting every x seconds

netfilter-mailcow-1  | # Warning: table ip filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | # Warning: table ip nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | # Warning: table ip6 filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | # Warning: table ip6 nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | Using NFTables backend
netfilter-mailcow-1  | Clearing all bans
netfilter-mailcow-1  | Initializing mailcow netfilter chain
netfilter-mailcow-1  | Setting MAILCOW isolation
netfilter-mailcow-1  | Watching Redis channel F2B_CHANNEL
netfilter-mailcow-1  | MAILCOW target is in position 7 in the ip forward table, restarting container to fix it...

I've not made changes to /etc/nftables.conf

[daanh432]

Can reproduce on Debian 10 and Ubuntu 22.04 using the steps described in original issue. The usage of ufw does not affect the end result.

Suggested workaround of settings DISABLE_NETFILTER_ISOLATION_RULE to Y has an effect as long as no SNAT has been set. If SNAT has been set it appears that there is also a incompatability on the nat table.

iptables v1.8.7 (nf_tables): table `nat' is incompatible, use 'nft' tool.

Mailcow's Netfilter seems to be breaking compatibility with nftables to iptables translation layer both for filters and network address translation.

[Dexus]

I had the same issue #5798 but was able on one of my servers, to fix it, with reoving old rules from /etc/rules.v{4,6} not the best option, but now i have build up all again and without the problems.

[milkmaker]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Fighter456]

If you encounter Problems, please try to set DISABLE_NETFILTER_ISOLATION_RULE=y in mailcow.conf and do a docker compose up -d. But beware that this might open a Vulnerability. To be on the safe side, restrict the access to the mailcow docker network with something like:

iptables: iptables -I DOCKER-USER ! -i br-mailcow -o br-mailcow -p tcp -m multiport --dport 3306,6379,8983,12345 -j DROP

nftables: nft insert rule ip "filter" "DOCKER-USER" iifname != "br-mailcow" oifname "br-mailcow" tcp dport {3306, 6379, 8983, 12345} counter packets 0 bytes 0 drop

Is this the official solution for this issue or is there something in the pipeline for an upcoming release?

Can a maintainer please clarify this?