Skip to content

kubedns container cannot connect to apiserver #193

New issue
New issue
Closed
Closed
kubedns container cannot connect to apiserver#193
@phagunbaya

Description

@phagunbaya
phagunbaya
opened on Mar 3, 2017

kubedns logs:

I0303 20:17:56.595813       1 dns.go:42] version: v1.6.0-alpha.0.680+3872cb93abf948-dirty
I0303 20:17:56.596373       1 server.go:107] Using https://10.96.0.1:443 for kubernetes master, kubernetes API: <nil>
I0303 20:17:56.596882       1 server.go:68] Using configuration read from ConfigMap: kube-system:kube-dns
I0303 20:17:56.596925       1 server.go:113] FLAG: --alsologtostderr="false"
I0303 20:17:56.596943       1 server.go:113] FLAG: --config-map="kube-dns"
I0303 20:17:56.596949       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0303 20:17:56.596952       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0303 20:17:56.596956       1 server.go:113] FLAG: --dns-port="10053"
I0303 20:17:56.596961       1 server.go:113] FLAG: --domain="cluster.local."
I0303 20:17:56.596967       1 server.go:113] FLAG: --federations=""
I0303 20:17:56.596971       1 server.go:113] FLAG: --healthz-port="8081"
I0303 20:17:56.596976       1 server.go:113] FLAG: --kube-master-url=""
I0303 20:17:56.596981       1 server.go:113] FLAG: --kubecfg-file=""
I0303 20:17:56.596985       1 server.go:113] FLAG: --log-backtrace-at=":0"
I0303 20:17:56.596992       1 server.go:113] FLAG: --log-dir=""
I0303 20:17:56.596996       1 server.go:113] FLAG: --log-flush-frequency="5s"
I0303 20:17:56.597001       1 server.go:113] FLAG: --logtostderr="true"
I0303 20:17:56.597005       1 server.go:113] FLAG: --stderrthreshold="2"
I0303 20:17:56.597009       1 server.go:113] FLAG: --v="2"
I0303 20:17:56.597014       1 server.go:113] FLAG: --version="false"
I0303 20:17:56.597019       1 server.go:113] FLAG: --vmodule=""
I0303 20:17:56.597113       1 server.go:155] Starting SkyDNS server (0.0.0.0:10053)
I0303 20:17:56.597414       1 server.go:165] Skydns metrics enabled (/metrics:10055)
I0303 20:17:56.597437       1 dns.go:144] Starting endpointsController
I0303 20:17:56.597443       1 dns.go:147] Starting serviceController
I0303 20:17:56.597531       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0303 20:17:56.597554       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
E0303 20:17:57.601223       1 sync.go:105] Error getting ConfigMap kube-system:kube-dns err: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/kube-dns: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:17:57.601271       1 dns.go:190] Error getting initial ConfigMap: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/kube-dns: dial tcp 10.96.0.1:443: getsockopt: no route to host, starting with default values
I0303 20:17:57.601317       1 dns.go:163] Waiting for Kubernetes service
I0303 20:17:57.601331       1 dns.go:169] Waiting for service: default/kubernetes
E0303 20:17:59.605100       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:01.607159       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:04.617151       1 reflector.go:199] pkg/dns/config/sync.go:114: Failed to list *api.ConfigMap: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dkube-dns&resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:05.613089       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:07.617099       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:09.619173       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:11.621183       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:15.629124       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:17.633140       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:19.635211       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:21.637134       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:25.645156       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E0303 20:18:26.598781       1 reflector.go:199] pkg/dns/dns.go:145: Failed to list *api.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

kube-apiserver logs

I0303 20:02:55.656265       1 config.go:527] Will report 10.160.20.150 as public IP address.
E0303 20:02:55.658840       1 reflector.go:199] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E0303 20:02:55.661056       1 reflector.go:199] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E0303 20:02:55.661974       1 reflector.go:199] k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:75: Failed to list *storage.StorageClass: Get http://127.0.0.1:8080/apis/storage.k8s.io/v1beta1/storageclasses?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E0303 20:02:55.662031       1 reflector.go:199] k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83: Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E0303 20:02:55.709032       1 reflector.go:199] pkg/controller/informers/factory.go:89: Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E0303 20:02:55.709152       1 reflector.go:199] pkg/controller/informers/factory.go:89: Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
[restful] 2017/03/03 20:02:55 log.go:30: [restful/swagger] listing is available at https://10.160.20.150:6443/swaggerapi/
[restful] 2017/03/03 20:02:55 log.go:30: [restful/swagger] https://10.160.20.150:6443/swaggerui/ is mapped to folder /swagger-ui/
I0303 20:02:55.771165       1 serve.go:88] Serving securely on 0.0.0.0:6443
I0303 20:02:55.771302       1 serve.go:102] Serving insecurely on 127.0.0.1:8080
I0303 20:02:56.730792       1 trace.go:61] Trace "Update /api/v1/namespaces/kube-system/pods/kube-apiserver-wyml01/status" (started 2017-03-03 20:02:55.825739356 +0000 UTC):
[63.968µs] [63.968µs] About to convert to expected version
[275.186µs] [211.218µs] Conversion done
[283.347µs] [8.161µs] About to store object in database
[904.938318ms] [904.654971ms] Object stored in database
[904.9425ms] [4.182µs] Self-link added
[905.006032ms] [63.532µs] END

Activity

phagunbaya

phagunbaya commented on Mar 3, 2017

@phagunbaya
phagunbaya
on Mar 3, 2017
Author

iptables

[root@wyml01 Falkonry-k8-installer]# iptables-save 
# Generated by iptables-save v1.4.21 on Fri Mar  3 13:23:40 2017
*nat
:PREROUTING ACCEPT [2:156]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-SAGRE6MUSU7ISKH2 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-SAGRE6MUSU7ISKH2 -s 10.160.20.150/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-SAGRE6MUSU7ISKH2 -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-SAGRE6MUSU7ISKH2 --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.160.20.150:6443
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-SAGRE6MUSU7ISKH2 --mask 255.255.255.255 --rsource -j KUBE-SEP-SAGRE6MUSU7ISKH2
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-SAGRE6MUSU7ISKH2
COMMIT
# Completed on Fri Mar  3 13:23:40 2017
# Generated by iptables-save v1.4.21 on Fri Mar  3 13:23:40 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8:452]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -j KUBE-FIREWALL
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -d 224.0.0.0/4 -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC-DEFAULT -m set --match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-iuZcey(5DeXbzgRFs8Szo]<@p dst -j ACCEPT
COMMIT
# Completed on Fri Mar  3 13:23:40 2017
pipejakob

pipejakob commented on Mar 7, 2017

@pipejakob
pipejakob
on Mar 7, 2017
Contributor

I suspect you're hitting issue #196. You can verify that this is the root cause by manually editing /etc/kubernetes/manifests/kube-apiserver.yaml on the master and changing the liveness probe:

livenessProbe:
  failureThreshold: 8
  httpGet:
    host: 127.0.0.1
    path: /healthz
    port: 443           # was 6443
    scheme: HTTPS
pipejakob

pipejakob commented on Mar 7, 2017

@pipejakob
pipejakob
on Mar 7, 2017
Contributor

@phagunbaya If you do try the above, I would also kill/restart kubelet for it to take effect faster. When I hit this problem myself, kubelet's exponential backoff was making it take forever to try to restart the kube-apiserver pod.

msavlani

msavlani commented on Mar 8, 2017

@msavlani
msavlani
on Mar 8, 2017

Did you try flushing your iptable rules and restart kubelet service ?

phagunbaya

phagunbaya commented on Mar 8, 2017

@phagunbaya
phagunbaya
on Mar 8, 2017
Author

@msavlani Flushing iptable rules did not help.
@pipejakob Thanks ! that resolved.

errordeveloper

errordeveloper commented on Apr 12, 2017

@errordeveloper
errordeveloper
on Apr 12, 2017

Also killing DNS pod seems to resolve this for me...

errordeveloper

errordeveloper commented on Apr 12, 2017

@errordeveloper
errordeveloper
on Apr 12, 2017

I am not entierly sure this has to do with #196, I think there is a race condition elsewhere. I've just hit this in something I'm working on at the moment, I will update if I figure out what causes it, as seem to have a way of reproducing is reliably.

TracyBin

TracyBin commented on Apr 14, 2017

@TracyBin
TracyBin
on Apr 14, 2017

I setup a single-machine Kubernetes cluster for development and faced the same problem.But modifying the port does not solve the problem

jeffchanjunwei

jeffchanjunwei commented on Apr 25, 2017

@jeffchanjunwei
jeffchanjunwei
on Apr 25, 2017

Hi @TracyBin, how do you solve this problem at last?

TracyBin

TracyBin commented on Apr 25, 2017

@TracyBin
TracyBin
on Apr 25, 2017 · edited by TracyBin

@jeffchanjunwei It is the problem of iptables.Please try the follow command

iptables -P FORWARD ACCEPT

If the command solve your problem,please tell me.

jeffchanjunwei

jeffchanjunwei commented on Apr 25, 2017

@jeffchanjunwei
jeffchanjunwei
on Apr 25, 2017

@TracyBin It doesn't work. kubedns-amd64:1.9 images still can not start. Errors as follows:

kubectl describe pod kubedns
<invalid> <invalid> 1 {kubelet k8sminion1} spec.containers{kubedns} Warning Unhealthy Readiness probe failed: Get http://10.233.124.95:8081/readiness: dial tcp 10.233.124.95:8081: getsockopt: connection refused

docker logs kubedns-amd
E0425 02:28:03.129272 1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: i/o timeout
E0425 02:28:03.234570 1 reflector.go:199] pkg/dns/dns.go:145: Failed to list *api.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: i/o timeout

pineking

pineking commented on May 12, 2017

@pineking
pineking
on May 12, 2017

@jeffchanjunwei do you solve this problem?

jeffchanjunwei

jeffchanjunwei commented on May 15, 2017

@jeffchanjunwei
jeffchanjunwei
on May 15, 2017

@pineking yes. It is the cause of network that results into the problem.

frankruizhi

frankruizhi commented on May 16, 2017

@frankruizhi
frankruizhi
on May 16, 2017 · edited by frankruizhi

I got the same issue,my kubedns log :

[root@k8s ~]# kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
I0516 07:38:31.041503 1 dns.go:42] version: v1.6.0-alpha.0.680+3872cb93abf948-dirty
I0516 07:38:31.042564 1 server.go:107] Using https://10.254.0.1:443 for kubernetes master, kubernetes API:
I0516 07:38:31.043625 1 server.go:68] Using configuration read from ConfigMap: kube-system:kube-dns
I0516 07:38:31.043729 1 server.go:113] FLAG: --alsologtostderr="false"
I0516 07:38:31.043762 1 server.go:113] FLAG: --config-map="kube-dns"
I0516 07:38:31.043773 1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0516 07:38:31.043780 1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0516 07:38:31.043786 1 server.go:113] FLAG: --dns-port="10053"
I0516 07:38:31.043810 1 server.go:113] FLAG: --domain="cluster.local."
I0516 07:38:31.043821 1 server.go:113] FLAG: --federations=""
I0516 07:38:31.043829 1 server.go:113] FLAG: --healthz-port="8081"
I0516 07:38:31.043837 1 server.go:113] FLAG: --kube-master-url=""
I0516 07:38:31.043847 1 server.go:113] FLAG: --kubecfg-file=""
I0516 07:38:31.043853 1 server.go:113] FLAG: --log-backtrace-at=":0"
I0516 07:38:31.043863 1 server.go:113] FLAG: --log-dir=""
I0516 07:38:31.043870 1 server.go:113] FLAG: --log-flush-frequency="5s"
I0516 07:38:31.043880 1 server.go:113] FLAG: --logtostderr="true"
I0516 07:38:31.043887 1 server.go:113] FLAG: --stderrthreshold="2"
I0516 07:38:31.043901 1 server.go:113] FLAG: --v="2"
I0516 07:38:31.043908 1 server.go:113] FLAG: --version="false"
I0516 07:38:31.043918 1 server.go:113] FLAG: --vmodule=""
I0516 07:38:31.044094 1 server.go:155] Starting SkyDNS server (0.0.0.0:10053)
I0516 07:38:31.048954 1 server.go:165] Skydns metrics enabled (/metrics:10055)
I0516 07:38:31.048987 1 dns.go:144] Starting endpointsController
I0516 07:38:31.048997 1 dns.go:147] Starting serviceController
I0516 07:38:31.049354 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0516 07:38:31.049383 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
E0516 07:39:01.051248 1 sync.go:105] Error getting ConfigMap kube-system:kube-dns err: Get https://10.254.0.1:443/api/v1/namespaces/kube-system/configmaps/kube-dns: dial tcp 10.254.0.1:443: i/o timeout
E0516 07:39:01.051307 1 dns.go:190] Error getting initial ConfigMap: Get https://10.254.0.1:443/api/v1/namespaces/kube-system/configmaps/kube-dns: dial tcp 10.254.0.1:443: i/o timeout, starting with default values
I0516 07:39:01.051338 1 dns.go:163] Waiting for Kubernetes service
I0516 07:39:01.051347 1 dns.go:169] Waiting for service: default/kubernetes
E0516 07:39:02.228560 1 reflector.go:199] pkg/dns/dns.go:145: Failed to list *api.Endpoints: Get https://10.254.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.254.0.1:443: i/o timeout
E0516 07:39:02.228646 1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.254.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.254.0.1:443: i/o timeout
E0516 07:39:32.229250 1 reflector.go:199] pkg/dns/config/sync.go:114: Failed to list *api.ConfigMap: Get https://10.254.0.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dkube-dns&resourceVersion=0: dial tcp 10.254.0.1:443: i/o timeout

I've tied a lot ,but none of them worked.

frankruizhi

frankruizhi commented on May 19, 2017

@frankruizhi
frankruizhi
on May 19, 2017 · edited by frankruizhi

I have found the solution to my problem:

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"a55267932d501b9fbd6d73e5ded47d79b5763ce5", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"a55267932d501b9fbd6d73e5ded47d79b5763ce5", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

1.First,we should make sure the ip-forward enabled on the linux kernel of every node.Just execute command:
sysctl net.ipv4.conf.all.forwarding = 1

2.Secondly,if your docker's version >=1.13,the default FORWARD chain policy was DROP,you should set default policy of the FORWARD chain to ACCEPT:$ sudo iptables -P FORWARD ACCEPT.

3.Then the configuration of the kube-proxy must be pass in :
--cluster-cidr=.

ps: --cluster-cidr string The CIDR range of pods in the cluster. It is used to bridge traffic coming from outside of the cluster. If not provided, no off-cluster bridging will be performed.
Refer to this:kubernetes/kubernetes#36835

5 remaining items

gogeof

gogeof commented on Aug 29, 2017

@gogeof
gogeof
on Aug 29, 2017 · edited by gogeof

I also have this problem in kubernetes v1.7.4, and after I restart docker, it fix.

BenHall

BenHall commented on Sep 5, 2017

@BenHall
BenHall
on Sep 5, 2017 · edited by BenHall

Also hitting this on a fair frequent basis with Kubernetes 1.7 on top of Docker 1.12.6

Running iptables -P FORWARD ACCEPT didn't resolve the issue.

luxas

luxas commented on Sep 5, 2017

@luxas
luxas
on Sep 5, 2017
Member

@BenHall please open a new issue with relevant details.

lastboy1228

lastboy1228 commented on Sep 17, 2017

@lastboy1228
lastboy1228
on Sep 17, 2017

systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker

The route problem can be solved by flush iptables.

BenHall
mentioned this on Sep 17, 2017
P4otocol

P4otocol commented on Oct 13, 2017

@P4otocol
P4otocol
on Oct 13, 2017

Thanks @frankruizhi for the info. Worked for me!! (Used docker version >1.13)

WanChengHu

WanChengHu commented on Oct 15, 2017

@WanChengHu
WanChengHu
on Oct 15, 2017

I got the same problem when I use kubeadm to init a k8s v1.8 cluster with one master and one node.

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

50 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @pineking@BenHall@errordeveloper@zxpower@ikus060

        Issue actions
          kubedns container cannot connect to apiserver · Issue #193 · kubernetes/kubeadm