No, kubernetes isn't listening to all interfaces by default. It picks the interface with the default gateway and listens to that. Use --api-advertise-addresses=<the eth1 ip addr> on kubeadm init in order to use the host-only interface.

cc @kubernetes/sig-cluster-lifecycle

To be clear, it does listen on all interfaces right now and we are planing to provide a flag to make it bind to a specific address, but default will probably remain 0.0.0.0 for now. Please also see #33562 (comment) and #33638.

Should be fixed in http://kubernetes.io/docs/getting-started-guides/kubeadm/#instructions now, @hgfischer could you please take a look and let me know if that docs change would have helped you out? Thanks!

What about the nodes themselves, why don't they have a flag to pass on to kubelet the --address flag (via kubeadm join)? I have a bare metal installation with two ethernet devices, it just so happens to be that my internal network doesn't have the gateway on it. When installing weave-net the IP used to communicate between nodes ends up becoming my external IP and not the internal one.

Thanks!

Hey @danielschonfeld could you please open a ticket against https://github.com/weaveworks/weave-kube for the latter issue? I'll ask my colleagues to take a look. Thanks!

@lukemarsden it's not a weave issue... weave only runs kube-peers which collects the advertised address from each node record from the API server. It's the kubelet on the individual nodes that does exactly what you described above. It takes the first IP with a gateway it finds and in my installation it just so happens to be that the first IP is the external one and not the internal one (cuz our internal network at our datacenter provider is with an unmanaged switch, meaning no gateway)

Ok, I see what's the problem. Node registration IP is outside of kubeadm.
Right now you will need to either add kubelet system drop-in unit, or
modify the one kubeadm install. Do know how to do this or you need more
details?

On Wed, 28 Sep 2016, 13:04 Daniel Schonfeld, notifications@github.com
wrote:

@lukemarsden https://github.com/lukemarsden it's not a weave issue...
weave only runs kube-peers which collects the advertised address from
each node record from the API server. It's the kubelet on the individual
nodes that does exactly what you described above. It takes the first IP
with a gateway it finds and in my installation it just so happens to be
that the first IP is the external one and not the internal one (cuz our
internal network at our datacenter provider is with an unmanaged switch,
meaning no gateway)

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#33618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWSy4xxjNrWYNVmoQXx5fK_z5rIsJXks5qulfDgaJpZM4KIRBs
.

@errordeveloper i'll need some details for the latter option. the former i'm familiar with, but i kinda like the latter idea better. Thank you!

The Debian and RPM packages of kubeadm install a drop-in under
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf, it has kubelet flags
needed for kubeadm to work. The file is the same on nodes and master. You
need to add a flag to this (IIRC --overide-external-hostname) and pass the
IP or hostname that you want, but check kubelet docs.

On Wed, 28 Sep 2016, 15:25 Daniel Schonfeld, notifications@github.com
wrote:

@errordeveloper https://github.com/errordeveloper i'll need some
details for the latter option. the former i'm familiar with, but i kinda
like the latter idea better. Thank you!

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#33618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWSxU8VaGwEHQ8OZaZf2SuU9ScQqWzks5quni9gaJpZM4KIRBs
.

@errordeveloper thank you!! i'll go tinker with it and report back if I run into trouble

@hgfischer this sounds like a more general bug (not so much related to kubeadm), could you please create another issue?

also does kubeadm join copy the 10-kubeadm.conf from the master on joining?

Missed this at first... The answer is - no, it doesn't need to go down to this level, in fact it's completely unaware of systemd as such. This file is installed by kubeadm package, and it's the same on master as well as all of the nodes.

Well i had the same issue in 1.7.1
Fixed it with:
kubeadm reset
kubeadm init --apiserver-advertise-address=10.10.10.4

Thank you luxas.

I know this thread is super long and pretty old but I figured I'd play my $0.02 for anyone else that comes along it. I also have many interfaces and struggle with kubeadm across various versions. I find that the best way to deploy my single-node for a local dev environment is to use a dummy interface. As you can't kubeadm on loopback (how annoying) and I've rarely been successful with default route, here is a tip for dummy dev. Note this is especially handy when using a laptop with only wifi as you can never expect to have consistent IP across access points and coffee shops and also can't use it when disconnected or on a plane without wifi:

$ sudo ip link add dummy0 type dummy

Or configure perm devices via here:
https://unix.stackexchange.com/questions/335284/how-can-we-create-multiple-dummy-interfaces-on-linux

Config a static IP on a dummy interface and use it for --apiserver-advertise-address. As there is nothing else messing with routes on that device, it's clean and ready to use locally. Super handy and sometimes bypasses issues with flannel/calico not getting routes from host (at least in my experience).

I'd like to add a comment that in order for me to get this to work (same issue as OP), it was necessary to add a route on the worker nodes:
ip route add 10.96.0.0/16 dev eth1 src <private network ip>
where "private network ip" is the ip assigned to the node's interface that's on the same network as the master node (likely a 172.28... address if you tend to use defaults in Vagrant).

The other solutions above were helpful in getting some but not all of the errors fixed. This ip route was the final piece needed.

Hi everyone,

Just to add to this, I was having problems getting my nodes to join the master node (this was not a cloud setup). I did an extra step which got everything working pretty well. I added an entry to my /etc/hosts file on the node itself that directed the hostname of the node to its IP address on the correct interface.

So for example, if your node's hostname is node-1.cluster.example.com and the IP address is 10.0.0.101 then you'd add the following line to your hosts file,

10.0.0.101 node-1.cluster.example.com

When you run the kubeadm join command, it should then use the correct interface.

@mwtzzz That solution worked, but now I am seeing multiple pods, running on different nodes with the same IP address. What might be wrong here?

Output of kubectl get pods -A -o wide:

NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE     IP          NODE       NOMINATED NODE   READINESS GATES
default       busybox-6cd57fd969-zr576           1/1     Running   0          3m16s   10.32.0.2   server-5   <none>           <none>
default       nginx-86c57db685-2zcjr             1/1     Running   0          11m     10.32.0.2   server-3   <none>           <none>
kube-system   coredns-6955765f44-gglvz           1/1     Running   5          3h31m   10.32.0.3   server-1   <none>           <none>
kube-system   coredns-6955765f44-nc8ff           1/1     Running   5          3h31m   10.32.0.2   server-1   <none>           <none>
kube-system   etcd-server-1                      1/1     Running   5          3h31m   10.0.2.15   server-1   <none>           <none>
kube-system   kube-apiserver-server-1            1/1     Running   5          3h31m   10.0.2.15   server-1   <none>           <none>
kube-system   kube-controller-manager-server-1   1/1     Running   5          3h31m   10.0.2.15   server-1   <none>           <none>
kube-system   kube-proxy-8mnlb                   1/1     Running   0          55m     10.0.2.15   server-5   <none>           <none>
kube-system   kube-proxy-9hpcr                   1/1     Running   1          55m     10.0.2.15   server-2   <none>           <none>
kube-system   kube-proxy-q2np4                   1/1     Running   0          55m     10.0.2.15   server-4   <none>           <none>
kube-system   kube-proxy-vw68k                   1/1     Running   1          56m     10.0.2.15   server-3   <none>           <none>
kube-system   kube-proxy-wjpbb                   1/1     Running   1          56m     10.0.2.15   server-1   <none>           <none>
kube-system   kube-scheduler-server-1            1/1     Running   5          3h31m   10.0.2.15   server-1   <none>           <none>
kube-system   weave-net-2jttm                    2/2     Running   28         158m    10.0.2.15   server-2   <none>           <none>
kube-system   weave-net-fhs59                    2/2     Running   27         158m    10.0.2.15   server-3   <none>           <none>
kube-system   weave-net-g67sp                    2/2     Running   9          158m    10.0.2.15   server-1   <none>           <none>
kube-system   weave-net-jqz69                    2/2     Running   30         158m    10.0.2.15   server-5   <none>           <none>
kube-system   weave-net-qc4ns                    2/2     Running   25         143m    10.0.2.15   server-4   <none>           <none>

Notice the first 4 lines!

Neither of the posted solutions are working for me with Packer/Vagrant and RHEL 7 using Kubeadm to bootstrap a v1.18.0 cluster.

Some network information:

LAN CIDR: 192.168.30.0/24
VPN CIDR: 172.27.0.0/16
Docker Engine CIDR: 192.168.65.0/24
Kubernetes Pod CIDR: 172.16.0.0/16
Kubernetes Service CIDR: 172.32.0.0/16
Calico IPV4 Pool CIDR: 172.16.0.0/16

The calico-node daemonset also has IP_AUTODETECTION_METHOD set to interface=eth1. This is because my Vagrant VMs come up with eth0 populated with 10.0.2.15/24. I'm not sure why. But eth0 has the proper IP Address supplied via Vagrant.

The calico-node pods running on each worker node are producing the following logs repeatedly:

[INFO][8] startup.go 365: Hit error connecting to datastore - retry error=Get https://172.32.0.1:443/api/v1/nodes/foo: dial tcp 172.32.0.1:443: connect: connection refused

I can curl that endpoint from the master node but not the worker nodes. Since they communicate via the LAN CIDR mentioned above: 192.168.30.0/24

Adding routes like @mwtzzz specified did not yield a working cluster unfortunately.

Any ideas?