Closed
Description
Kubernetes version (use kubectl version): v1.4.6, v1.5.1, likely many versions
Environment:
- Cloud provider or hardware configuration: Azure / Azure Container Service
- OS (e.g. from /etc/os-release): Ubuntu Xenial
- Kernel (e.g.
uname -a): latest 16.04-LTS kernel - Install tools: Cloud-Init + hyperkube
- Others:
Configuration Details:
kubeletruns in a container- master services run as static manifests
- kube-addon-manager runs as a static manifest
kube-proxyruns in iptables mode via a daemonset
What happened:
After upgrading to docker 1.13.0 on the nodes, outbound container traffic stops working
What you expected to happen:
Outbound container traffic to work (aka, I can hit the internet and service ips from inside the container)
How to reproduce it (as minimally and precisely as possible):
Deploy an ACS Kubernets cluster. If the workaround has rolled out, then force upgrade docker to 1.13 (you'll have to remove a pin we're setting in /etc/apt/preferences.d).
Unclear if this repros on other configurations right now.
Anything else do we need to know:
No, I just don't know where/how to best troubleshoot this.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Participants
Activity
0xmichalis commentedon Jan 20, 2017
@kubernetes/sig-node-misc
dkerwin commentedon Jan 23, 2017
Can confirm the problem with k8s 1.4.7 & docker 1.13 on debian jessie. kubelet managed by systemd
colemickens commentedon Jan 24, 2017
Since the team @Kargakis tagged here is no longer a team... cc: @kubernetes/sig-node-bugs
Packages: pin docker version to 1.12.6 due to kubernetes/kubernetes#4…
bboreham commentedon Jan 31, 2017
Docker 1.13 changed the default iptables forwarding policy to DROP, which has effects like this.
You can change the policy to ACCEPT (which it was in Docker 1.12 and before) by running:
on every node. You need to run this in the host network namespace, not inside a pod namespace.
MaesterZ commentedon Jan 31, 2017
Tested out and working
Environment:
feiskyer commentedon Feb 1, 2017
Could someone explain why docker defaulting to DROP? Does this mean containers of docker v1.13 can't connect outside by default?
bboreham commentedon Feb 1, 2017
@feiskyer generally the Linux default is to have IP forwarding off.
Docker used to turn it on across the board, which was (a) unnecessary and (b) a security issue. 1.13 removed this issue.
Docker add two specific rules which allow traffic off their bridge, and replies to come back:
CNI providers which do not use the
docker0bridge need to make similar provision.bboreham commentedon Feb 1, 2017
@colemickens can you clarify which network plugin you are using - is it
kubenet?Karimerto commentedon Feb 1, 2017
This thread sure was a lifesaver, though sadly I found it about 14 hours too late.. I managed to wipe my entire cluster and reinstall everything, with the same issue still persisting. I was about to lose my mind trying to figure out why half of my original cluster was working and the other wasn't. Those which didn't work were installed and added later with docker 1.13, so this explains everything.
Now I've got everything up and running again!
Thanks again for this 👍
jbeda commentedon Feb 2, 2017
The docker change that caused this: moby/moby#28257
colemickens commentedon Feb 2, 2017
(@bboreham Yes, it was
kubenet.)80 remaining items