Yes this happens to me too on ubuntu when I re-setup my firewall rules with iptable (ansible raw_iptables). I then need to restart docker daemon to get them back completely.

it would be very nice if there would be a docker reload iptables for to get the iptables back into the system iptables. Restarting the docker daemon everytime I re-setup the firewall rules is not a nice option.

Strange, it does not happen to me with docker-ce-17.06.0, CentOs 7.3. Reloading firewalld has no impact on docker rules. Maybe a regression of docker-ce?

I have the same machine as OP. I'm also having this same issue.
I could not see this behavior on a similar machine (CentOS 7.2 and Docker-CE v17).
Is this related to how firewalld manages rules, or to how Docker saves rules to iptables?

I also have been hunt this down for a while and here is what seems to be working for me:

TLDR first; If you add the DOCKER-USER chain to firewalld so that it is present in iptables before docker starts then you should be able to apply rules.

Longer story; Firewalld and Docker both use iptables to route traffic. Firewalld always flushes iptables rules and only reinstates rules that have been configured with firewalld. Docker, when it starts, adds a number of chains to the iptables that can possibly conflict with your rules. However, Docker does respect a special "DOCKER-USER" chain that you can configure to filter traffic. If the "DOCKER-USER" chain is not present when Docker starts, Docker will add it and allow all connections being passed to it. However, if the DOCKER-USER chain already exists, it will not do anything to it (except add an "ACCEPT ALL rule" to the END of the chain (which won't do anything if you configure the previous rules in the chain cover all traffic cases).

Here is a sequence of commands I have used over and over again to get my firewalld settings correct. My goal was to only allow traffic to http and https from a few ip addresses:

# Removing DOCKER-USER CHAIN (it won't exist at first)
firewall-cmd --permanent --direct --remove-chain ipv4 filter DOCKER-USER

# Flush rules from DOCKER-USER chain (again, these won't exist at first; firewalld seems to remember these even if the chain is gone)
firewall-cmd --permanent --direct --remove-rules ipv4 filter DOCKER-USER

# Add the DOCKER-USER chain to firewalld
firewall-cmd --permanent --direct --add-chain ipv4 filter DOCKER-USER

# Add rules (see comments for details)
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -m comment --comment "This allows docker containers to connect to the outside world"
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -j RETURN -s 172.18.0.0/16 -m comment --comment "allow internal docker communication"
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -p tcp -m multiport --dports https -s 123.456.7.89/32 -j ACCEPT -m comment --comment "my allowed ip address to http and https ports"

#Add as many ip or other rules and then run this command to block all other traffic
firewall-cmd --permanent --direct --add-rule ipv4 filter DOCKER-USER 0 -j REJECT -m comment --comment "reject all other traffic"

# restart the services 
systemctl stop docker
systemctl stop firewalld
systemctl start firewalld
systemctl start docker

@brianbolt

I have a problem with the fact that Docker devs haven't commented on this issue report. Even though your solution works now, it might not work in the future if the CentOS and Docker devs don't agree on the correct method on doing this.

I think the main issue here is that docker does not manager the DOCKER-USER chain.

Perhaps it should re-add the chain when firewalld is reloaded but none of the rules in that chain will be reloaded because these aren't docker's rules.

I think @brianbolt's response is exactly the correct way to deal with this since they are user defined rules.

@cpuguy83

Do you mean the DOCKER-USER chain is created by some CentOS service and not by the docker engine?

@aki-k The DOCKER-USER chain itself is created by docker along with the main jump rule, but that is all. The chain is expected to be managed by users.

I'll open a PR to ensure that the chain and the initial rule is re-created on firewalld reload.

opened moby/libnetwork#2053 to ensure the chain is re-created on firewalld reload, but again any user rules will need to be reloaded by the user.

Seems like @brianbolt 's solution is still working on centos 8 stream with

[root@holod files]# docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:02:19 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:24 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0