track/close kubelet->API connections on heartbeat failure #63492

liggitt · 2018-05-07T16:17:39Z

xref #48638
xref kubernetes-retired/kube-aws#598

we're already typically tracking kubelet -> API connections and have the ability to force close them as part of client cert rotation. if we do that tracking unconditionally, we gain the ability to also force close connections on heartbeat failure as well. it's a big hammer (means reestablishing pod watches, etc), but so is having all your pods evicted because you didn't heartbeat.

this intentionally does minimal refactoring/extraction of the cert connection tracking transport in case we want to backport this

first commit unconditionally sets up the connection-tracking dialer, and moves all the cert management logic inside an if-block that gets skipped if no certificate manager is provided (view with whitespace ignored to see what actually changed)
second commit plumbs the connection-closing function to the heartbeat loop and calls it on repeated failures

follow-ups:

consider backporting this to 1.10, 1.9, 1.8
refactor the connection managing dialer to not be so tightly bound to the client certificate management

/sig node
/sig api-machinery

kubelet: fix hangs in updating Node status after network interruptions/changes between the kubelet and API server

liggitt · 2018-05-07T16:22:33Z

@kubernetes/sig-node-pr-reviews

liggitt · 2018-05-07T16:41:00Z

cc @sjenning

awly

Recommend adding a test to make sure OnHeartbeatFailure triggers.

awly · 2018-05-07T16:37:39Z

cmd/kubelet/app/server.go

@@ -541,19 +541,19 @@ func run(s *options.KubeletServer, kubeDeps *kubelet.Dependencies) (err error) {
 			return fmt.Errorf("invalid kubeconfig: %v", err)
 		}

-		var clientCertificateManager certificate.Manager
+		var clientCertificateManager certificate.Manager = nil


This doesn't seem to do anything useful, remove = nil

awly · 2018-05-07T16:39:07Z

pkg/kubelet/certificate/transport.go

@@ -51,80 +51,93 @@ import (
 //
 // stopCh should be used to indicate when the transport is unused and doesn't need
 // to continue checking the manager.
-func UpdateTransport(stopCh <-chan struct{}, clientConfig *restclient.Config, clientCertificateManager certificate.Manager, exitAfter time.Duration) error {
+func UpdateTransport(stopCh <-chan struct{}, clientConfig *restclient.Config, clientCertificateManager certificate.Manager, exitAfter time.Duration) (func(), error) {


Document the new return value in func comment

liggitt · 2018-05-07T17:10:46Z

Recommend adding a test to make sure OnHeartbeatFailure triggers.

done

dims · 2018-05-07T18:34:09Z

pkg/kubelet/kubelet_node_status.go

@@ -350,6 +350,9 @@ func (kl *Kubelet) updateNodeStatus() error {
 	glog.V(5).Infof("Updating node status")
 	for i := 0; i < nodeStatusUpdateRetry; i++ {
 		if err := kl.tryUpdateNodeStatus(i); err != nil {
+			if i > 0 && kl.onRepeatedHeartbeatFailure != nil {
+				kl.onRepeatedHeartbeatFailure()


Do we want to set onRepeatedHeartbeatFailure to nil or something? (once we invoke the method)

No, that would mean the kubelet would hit the same issue if the network condition was encountered twice during a single process lifetime

Ack thanks!

liggitt · 2018-05-07T19:41:17Z

/test pull-kubernetes-e2e-gke

liggitt · 2018-05-07T19:41:29Z

/test pull-kubernetes-local-e2e

k8s-github-robot · 2018-05-14T23:08:12Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-05-14T23:56:35Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

liggitt · 2018-05-14T23:59:22Z

I plan to open picks to 1.8, 1.9, and 1.10, but will hold them until this makes it through serial/soak/scale CI tests

liggitt · 2018-05-15T20:39:22Z

green serial run containing this PR:
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/3477

many green slow suite CI runs containing this PR, e.g.:
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/17078

scale test showed no regressions (https://k8s-testgrid.appspot.com/sig-release-master-blocking#gce-scale-performance, run 154)

…2-upstream-release-1.10 Automatic merge from submit-queue. Automated cherry pick of #63492: Always track kubelet -> API connections Cherry pick of #63492 on release-1.10. #63492: Always track kubelet -> API connections

…2-upstream-release-1.8 Automatic merge from submit-queue. Automated cherry pick of #63492: Always track kubelet -> API connections Cherry pick of #63492 on release-1.8. #63492: Always track kubelet -> API connections

…2-upstream-release-1.9 Automatic merge from submit-queue. Automated cherry pick of #63492: Always track kubelet -> API connections Cherry pick of #63492 on release-1.9. #63492: Always track kubelet -> API connections

k8s-ci-robot requested review from awly, dims and yifan-gu May 7, 2018 16:17

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels May 7, 2018

liggitt mentioned this pull request May 7, 2018

kubelet fails to heartbeat with API server with stuck TCP connections #48638

Closed

liggitt force-pushed the node-heartbeat-close-connections branch from 60fd863 to 05116f9 Compare May 7, 2018 16:40

awly suggested changes May 7, 2018

View reviewed changes

liggitt force-pushed the node-heartbeat-close-connections branch 2 times, most recently from 8a0a991 to c47a7a9 Compare May 7, 2018 17:10

liggitt changed the title ~~WIP - track/close kubelet->API connections on heartbeat failure~~ track/close kubelet->API connections on heartbeat failure May 7, 2018

liggitt force-pushed the node-heartbeat-close-connections branch from c47a7a9 to f18a52d Compare May 7, 2018 17:33

dims reviewed May 7, 2018

View reviewed changes

liggitt changed the title ~~track/close kubelet->API connections on heartbeat failure~~ WIP - track/close kubelet->API connections on heartbeat failure May 7, 2018

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 7, 2018

liggitt added 2 commits May 7, 2018 15:06

Always track kubelet -> API connections

52876f7

Close all kubelet->API connections on heartbeat failure

814b065

liggitt force-pushed the node-heartbeat-close-connections branch from f18a52d to 814b065 Compare May 7, 2018 19:07

k8s-github-robot merged commit 8220171 into kubernetes:master May 14, 2018

liggitt deleted the node-heartbeat-close-connections branch May 15, 2018 00:10

This was referenced May 15, 2018

Automated cherry pick of #63492: Always track kubelet -> API connections #63831

Merged

Automated cherry pick of #63492: Always track kubelet -> API connections #63832

Merged

Automated cherry pick of #63492: Always track kubelet -> API connections #63834

Merged

zachaller mentioned this pull request May 17, 2018

api: loadbalancer: Internal does not create correct Route53 entries kubernetes/kops#4252

Closed

redbaron mentioned this pull request May 25, 2018

Kubelet: Make heartbeat logic explicit. #45141

Closed

mattatcha mentioned this pull request Jun 25, 2018

Rolling update puts nodes into "not ready" kubernetes/kops#4946

Closed

nevermore-muyi mentioned this pull request Jul 12, 2018

高可用apiserver中，Node短暂出现Not Ready easzlab/kubeasz#257

Closed

liggitt mentioned this pull request Jul 19, 2018

Update http.Transport if it already exists in ExecProvider #66395

Merged

MariusDieckmann mentioned this pull request Jul 23, 2018

Worker nodes randomly become inactive rancher/rancher#13382

Closed

This was referenced Jul 31, 2018

TCP user timeout for Kubelet ↔️ apiserver connection #48670

Closed

Unable to detect if a watch is active #65012

Closed

kanantheswaran-splunk mentioned this pull request Aug 3, 2018

Master reboot can cause etcd/ etcd-events pods to not start up because of races kubernetes/kops#5578

Closed

jackfrancis added a commit to jackfrancis/aks-engine that referenced this pull request Jan 3, 2019

fixed in kubernetes/kubernetes#63492

b63ef19

jackfrancis mentioned this pull request Jan 3, 2019

docs: remove obsolete "known issue" Azure/aks-engine#225

Merged

4 tasks

jackfrancis added a commit to Azure/aks-engine that referenced this pull request Jan 3, 2019

fixed in kubernetes/kubernetes#63492 (#225)

8172106

arslanbekov mentioned this pull request Jan 18, 2019

kubelet stopped working unexpectedly #57955

Closed

juhacket pushed a commit to juhacket/aks-engine that referenced this pull request Mar 14, 2019

fixed in kubernetes/kubernetes#63492 (Azure#225)

8b90428

liggitt mentioned this pull request Mar 29, 2019

kubelet stopped posting node status, seems to be hung #75846

Closed

gaorong mentioned this pull request May 17, 2019

kubelet: fix fail to close kubelet->API connections on heartbeat failure #78016

Merged

paulgmiller mentioned this pull request Jun 13, 2019

Kube-proxy hangs on api-server errors #76338

Closed

aojea mentioned this pull request Feb 14, 2022

kubelet apiserver: be gentle closing connections on heartbeat failures #108107

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

track/close kubelet->API connections on heartbeat failure #63492

track/close kubelet->API connections on heartbeat failure #63492

liggitt commented May 7, 2018 •

edited

liggitt commented May 7, 2018

liggitt commented May 7, 2018

awly left a comment

awly May 7, 2018

awly May 7, 2018

liggitt May 7, 2018

liggitt commented May 7, 2018

dims May 7, 2018

liggitt May 7, 2018

dims May 7, 2018

liggitt commented May 7, 2018

liggitt commented May 7, 2018

k8s-github-robot commented May 14, 2018

k8s-github-robot commented May 14, 2018

liggitt commented May 14, 2018

liggitt commented May 15, 2018 •

edited

track/close kubelet->API connections on heartbeat failure #63492

track/close kubelet->API connections on heartbeat failure #63492

Conversation

liggitt commented May 7, 2018 • edited

liggitt commented May 7, 2018

liggitt commented May 7, 2018

awly left a comment

Choose a reason for hiding this comment

awly May 7, 2018

Choose a reason for hiding this comment

awly May 7, 2018

Choose a reason for hiding this comment

liggitt May 7, 2018

Choose a reason for hiding this comment

liggitt commented May 7, 2018

dims May 7, 2018

Choose a reason for hiding this comment

liggitt May 7, 2018

Choose a reason for hiding this comment

dims May 7, 2018

Choose a reason for hiding this comment

liggitt commented May 7, 2018

liggitt commented May 7, 2018

k8s-github-robot commented May 14, 2018

k8s-github-robot commented May 14, 2018

liggitt commented May 14, 2018

liggitt commented May 15, 2018 • edited

liggitt commented May 7, 2018 •

edited

liggitt commented May 15, 2018 •

edited