Thank you for the excellent report!

So my theory is this:

1. CLOSE_WAIT happens when we get a TCP reset from the other end of the connection, but we don't close the socket
2. For normal kube-proxy traffic using iptables, this won't happen - it is iptables packet-manipulation, not a "real connection"
3. But when there are no pods in a service, the nodeport iptables rules are simply not there; connections to the nodeport are not rewritten.
4. So during this time period, connections go to kube-proxy on the NodePort. kube-proxy is listening on the NodePort (because it wants to make sure the port is not in use), at https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1349 But kube-proxy never actually does anything with the socket. Technically we don't even accept the connections, and we certainly don't actively close them.
5. Connections that hit the kube-proxy nodeport socket will be in CLOSE_WAIT until kube-proxy is restarted; effectively forever on a healthy node.

For 3: I confirmed that during the time a service had no pods, the KUBE-SVC-UV4XIKEQLMZEPCEV (in my case) was removed entirely...

-A KUBE-NODEPORTS -p tcp -m comment --comment "default/ingress-nginx:http" -m tcp --dport 31445 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/ingress-nginx:http" -m tcp --dport 31445 -j KUBE-SVC-UV4XIKEQLMZEPCEV

And kube-proxy was listening on 31445 (the NodePort):

tcp6       0      0 :::31445                :::*                    LISTEN      1333/kube-proxy 
tcp6      53      0 172.20.122.50:31445     172.20.108.40:38369     CLOSE_WAIT  -               
tcp6      54      0 172.20.122.50:31445     172.20.104.161:2100     CLOSE_WAIT  -               
tcp6      54      0 172.20.122.50:31445     172.20.104.161:2092     CLOSE_WAIT  -               
tcp6      55      0 172.20.122.50:31445     172.20.104.161:44177    CLOSE_WAIT  -               
tcp6      53      0 172.20.122.50:31445     172.20.108.40:16199     CLOSE_WAIT  -               

Also, the number of CLOSE_WAIT connections went up during the time period when I restarted the pod in my service.

I confirmed that 172.20.104.161 and 172.20.108.40 are the IP addresses of my ELBs. They are doing TCP health checks every 5s (IIRC). It is also possible that the health check is an unusual TCP pattern (because it is not an HTTP health check; it merely opens and closes the connection).

For this particular issue, which was about the master, my suspicion is that the same will happen on the nodes, in that the kube-proxy configurations should be the same. If we actually know that this does not happen on the nodes, that is interesting information.

Two possible fixes spring to mind:

A) Add a rule when there are no Pods on the NodePort that rejects the connection. Efficient. But: iptables is never easy. Also I don't know if this will cause health checks to fail, which isn't wrong but would slow down recovery.
B) Have kube-proxy accept connections and immediately close them. This feels correct, though I think it will consume a goroutine per nodeport. But goroutines are cheap.

cc @felipejfc as this looks similar to what you are reporting in #41640

cc @thockin for kube-proxy guru-ness and advice on which option to pursue

I actually had 2 namespaces with 3 services each(ELB type) that had no pods associated because someone forgotten do delete them after deleting the pods and they had healthchecks configured, after deleting the services today we've seen a massive networking performance boost (thousands of sockets in CLOSE_WAIT state got closed).

I'll look for other services with no pod associated and delete them and will keep an eye on the cluster

Thanks for helping @justinsb !!

@justinsb do you recall if we ported this back to 1.6?

@thockin looks like we got the first one into 1.6, but not the second one :-(

These are the two commits (for some reason github only shows the branches in this view, not the PR view...)

2ec8799

9a423b6

I did reopen the cherry-pick for the 1st to 1.5 this morning (I've been getting pings on this issue): #43858

Looks like we should get #43858 in, and then cherry-pick 9a423b6 to 1.5 and 1.6.

I do recommend to people hitting this in the real world that they remove services without endpoints - it is almost always just an error/oversight. I don't think it's a huge problem to leak a few connections on a restart if you happen to end up with no pods for a ~minute. Also, typically this saves the cost of an extra ELB.

What version of Kubernetes is it expected that this issue is resolved in? The problem still manifests on my Kubernetes 1.6.3 deployment.

still on kubernetes 1.6.4 on aws, service which type is load balancer in our cluster are all pods-associated, but still have thousands of CLOSE_WAIT