Sounds like you've removed your ingress network.
You can recreate it with docker network create --ingress --driver overlay ingress

I have not deleted any network. When I did "docker network ls" ingress was there in the list.

Is it possible you ran docker system prune when no services were running?

Could you provide

• the output of docker network ls
• the output of docker network inspect ingress

Does it work after you re-create the ingress network (using the steps @cpuguy83 provided?)

After getting above issue, I ran docker network prune which removed ingress network.
Later, I recreated ingress network and it worked fine.

Ok, I suspect what happened, is that this was a version of docker that was upgraded from an older version. Older versions marked "ingress" networks in a different way, and there was a bug in 17.05 which prevented docker from recognising such networks. As a result, docker network prune (or docker system prune) removed the ingress network if no container / service was attached to it.

That situation will be resolved in docker 17.06 (see #33286). If I'm correct, an ingress network created with docker 17.05 will not be removed by docker network prune, so if you would run docker network prune again, the network should not be removed.

I'll go ahead and close this issue, because this will be addressed in docker 17.06, but feel free to continue the conversation 👍

@thaJeztah Will try network prune.

I am still confused which version of docker to use. In any of the docker versions after 1.12 some thing will break.

Could you please let me know which is the most stable version for use in production for docker swarm mode?

In any of the docker versions after 1.12 some thing will break.

Can you provide information what breaks?

Could you please let me know which is the most stable version for use in production for docker swarm mode?

The 17.03.x and 17.06.x (to be released soon), are "quarterly" releases, which have a 4-month support cycle for critical bug fixes (community edition), and are supported for 1 year (enterprise edition)

Can you provide information what breaks?

• In 1.13, I faced issues with selinux SELinux relabeling of NFS volumes is not allowed since 1.12.3 #31255.
• In 17.05-ce I faced above issue Error response from daemon: rpc error: code = 9 desc = service needs ingress network, but no ingress network is present #33420.

Will try out 17.06. Hope we can use it in prod :)

Thanks for the help.

#31255 actually is an issue with docker 1.12.4. Note that Docker 17.03.0 is actually what 1.13.2 would have been, so no functional changes from 1.13.0, only bug fixes. For production, I'd recommend installing the "quarterly" releases, unless upgrading once a month is no problem for your situation. For longer support cycles, Docker EE provides 1 year of support (bug fixes and security fixes), so may be worth considering for production

For longer support cycles, Docker EE provides 1 year of support (bug fixes and security fixes), so may be worth considering for production

Will look into this.

@thaJeztah can confirm your suspicion here, just ran into the same issue on my dev environment: installed a swarm in the past, upgraded Docker and ran docker system prune while the swarm was offline, brought the swarm back up and got this. docker network create --ingress --driver overlay ingress resolved the issue.

The method @cabloo suggested can remove the problem. It still happens in 17.06.0-ce.

@thaJeztah: I'm having this issue upgrading our swarm from 17.09 to 17.12. All the nodes are now 17.12, but I can't seem to rebuild the ingress network.

$ docker service create --name=swarm-visualizer \
  --publish=8080:8080/tcp \
  --constraint=node.role==manager \
  --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
  dockersamples/visualizer

Error response from daemon: rpc error: code = FailedPrecondition desc = service needs ingress network, but no ingress network is present

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
68bb9e73c8db        bridge              bridge              local
315e6882961f        docker_gwbridge     bridge              local
d972764776bb        host                host                local
2aittxoqcj4c        ingress             overlay             swarm
64614b19d7aa        none                null                local
js7axcrq1ddj        proxy               overlay             swarm
whthoj3zcugc        swarmpit_net        overlay             swarm

$ docker network rm ingress
WARNING! Before removing the routing-mesh network, make sure all the nodes in your swarm run the same docker engine version. Otherwise, removal may not be effective and functionality of newly create ingress networks will be impaired.
Are you sure you want to continue? [y/N] y
ingress

$ docker network create --ingress --driver overlay ingress
Error response from daemon: network with name ingress already exist

So, it doesn't really remove the ingress network.

@johnjelinek could it be there were still tasks lingering around that were using the ingress network? Anything in the logs that could be helpful?

nah, previously when I ran docker network rm ingress, it would complain about a service that depended on it (like swarm-visualizer). So I removed all those services first, and now I'm trying to docker network rm ingress, and it won't go away, and yet I can't bring the swarm-visualizer back.

I saw these errors in the logs, could that be related?

Feb 07 18:54:22 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:22.661441266Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 07 18:54:23 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:23.908867756Z" level=info msg="memberlist: Suspect 2a3ee67b8d68 has failed, no acks received"
Feb 07 18:54:48 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:48.005063849Z" level=info msg="NetworkDB stats dfw-dev-swm-m-1(8cafedcba7b1) - netID:2aittxoqcj4cne658awt4q596 leaving:false netPeers:2 entries:4 Queue qLen:0 n
Feb 07 18:54:48 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:48.005700612Z" level=info msg="NetworkDB stats dfw-dev-swm-m-1(8cafedcba7b1) - netID:whthoj3zcugc5qq9trcueqz8y leaving:false netPeers:5 entries:12 Queue qLen:0
Feb 07 18:54:50 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:50.646988639Z" level=info msg="Node 7681dc520428/172.20.96.132, joined gossip cluster"
Feb 07 18:54:50 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:50.647536681Z" level=info msg="Node 7681dc520428/172.20.96.132, is the new incarnation of the failed node 2a3ee67b8d68/172.20.96.132"
Feb 07 18:54:50 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:50.647830092Z" level=info msg="Node 2a3ee67b8d68 change state NodeFailed --> NodeLeft"
Feb 07 18:54:50 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:54:50.648100530Z" level=info msg="Node 7681dc520428/172.20.96.132, added to nodes list"
Feb 07 18:55:00 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:55:00.148770834Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 07 18:55:00 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:55:00.149299708Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"
Feb 07 18:55:00 dfw-dev-swm-m-1 dockerd[14070]: time="2018-02-07T18:55:00.149556987Z" level=error msg="Failed to deserialize netlink ndmsg: Link not found"

meh, ended up resorting to blowing away the swarm and starting from scratch 😖 ingress network is back on all nodes now.

You could have just restarted the docker daemon /: -several months later in the same shoe but with more luck. The thing is remove works, but somehow stays :) - so you have to restart docker to realise the fact, that it's no more :)

I am having the same problem, in that I have a ingress (overlay) network, and need to publish a extra port on the ingress service on that network.

# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
d8433bd0630a        bridge              bridge              local
86bef805c901        docker_gwbridge     bridge              local
6a5481d980be        host                host                local
8ia1io3kwjzl        ingress_ingress     overlay             swarm
a7f1f68d6c4e        none                null                local
# docker service inspect --format="{{json .Endpoint.Spec.Ports}}" ingress_ingress | sed 's/},{/&\n/g'
[{"Protocol":"tcp","TargetPort":80,"PublishedPort":80,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":443,"PublishedPort":443,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":3000,"PublishedPort":3000,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":8080,"PublishedPort":8080,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":8443,"PublishedPort":8443,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":8888,"PublishedPort":8888,"PublishMode":"host"}]
# docker --version
Docker version 18.09.5, build e8ff056

But when I try publish a extra port, I get the same error...

# docker service update --publish-add 3001  ingress_ingress 
Error response from daemon: rpc error: code = FailedPrecondition desc = service needs ingress network, but no ingress network is present

Services using the network are active so I can not

• shutdown and restart the system,
• nor remove the network

as previously suggested @johnjelinek

The ONLY solution I have found so far is to manually route the port with iptable entries to the IP of the service, basically doing what docker should be doing...

  iptables -t nat -A DOCKER ! -i docker_gwbridge -p tcp -m tcp \
           --dport 3001 -j DNAT --to-destination 172.18.0.2:3001
  iptables -t nat -A POSTROUTING -j MASQUERADE -p tcp \
           --source 172.18.0.2 --destination 172.18.0.2 --dport 3001
  iptables -A DOCKER -j ACCEPT -p tcp --destination 172.18.0.2 \
           \! -i docker_gwbridge -o docker_gwbridge --dport 3001

And this worked... But of course docker does NOT know about it so when the service was restarted, the above route disappears and requires manual re-creation. Not an ideal solution.

Any other solutions?

I am having the same problem, in that I have a ingress (overlay) network, and need to publish a extra port on the ingress service on that network.

# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
d8433bd0630a        bridge              bridge              local
86bef805c901        docker_gwbridge     bridge              local
6a5481d980be        host                host                local
8ia1io3kwjzl        ingress_ingress     overlay             swarm
a7f1f68d6c4e        none                null                local
# docker service inspect --format="{{json .Endpoint.Spec.Ports}}" ingress_ingress | sed 's/},{/&\n/g'
[{"Protocol":"tcp","TargetPort":80,"PublishedPort":80,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":443,"PublishedPort":443,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":3000,"PublishedPort":3000,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":8080,"PublishedPort":8080,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":8443,"PublishedPort":8443,"PublishMode":"host"},{
"Protocol":"tcp","TargetPort":8888,"PublishedPort":8888,"PublishMode":"host"}]
# docker --version
Docker version 18.09.5, build e8ff056

But when I try publish a extra port, I get the same error...

# docker service update --publish-add 3001  ingress_ingress 
Error response from daemon: rpc error: code = FailedPrecondition desc = service needs ingress network, but no ingress network is present

Services using the network are active so I can not

• shutdown and restart the system,
• nor remove the network

as previously suggested @johnjelinek

The ONLY solution I have found so far is to manually route the port with iptable entries to the IP of the service, basically doing what docker should be doing...

  iptables -t nat -A DOCKER ! -i docker_gwbridge -p tcp -m tcp \
           --dport 3001 -j DNAT --to-destination 172.18.0.2:3001
  iptables -t nat -A POSTROUTING -j MASQUERADE -p tcp \
           --source 172.18.0.2 --destination 172.18.0.2 --dport 3001
  iptables -A DOCKER -j ACCEPT -p tcp --destination 172.18.0.2 \
           \! -i docker_gwbridge -o docker_gwbridge --dport 3001

And this worked... But of course docker does NOT know about it so when the service was restarted, the above route disappears and requires manual re-creation. Not an ideal solution.

Any other solutions?

As far as I know, docker network prune and recreate the ingress network will be working. But this can be incorrect information thus I quit using docker on 2017.

docker network prune did not do anything, and I would not expect it to, as there are still services using the network!

Okay was able to get service to restart (shutdown down everything... It was just lucky situation where users were suddenly not using the system (a rare occurance).. Everything was started up fine though the 'manual routing' hack I was using before had to also be manually removed to get things working properly (just stopped connections).

Still it would be better if --publish-add actually could work for services with an attached ingress network. I really would not want to go though the hoops I have had to endure to add ONE extra port.