StatefulSet - can't rollback from a broken state #67250

MrTrustor · 2018-08-10T10:21:30Z

/kind bug

What happened:

I updated a StatefulSet with a non-existent Docker image. As expected, a pod of the statefulset is destroyed and can't be recreated (ErrImagePull). However, when I change back the StatefulSet with an existing image, the StatefulSet doesn't try to remove the broken pod to replace it by a good one. It keeps trying to pull the non-existing image.
You have to delete the broken pod manually to unblock the situation.

Related Stackoverflow question

What you expected to happen:

When rolling back the bad config, I expected the StatefulSet to remove the broken pod and replace it by a good one.

How to reproduce it (as minimally and precisely as possible):

Deploy the following StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: "nginx"
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 10Gi

Once the 3 pods are running, update the StatefulSet spec and change the image to k8s.gcr.io/nginx-slim:foobar
Observe the new pod failing to pull the image.
Roll back the change.
Observe the broken pod not being deleted.

Anything else we need to know?:

I observed this behaviour both on 1.8 and 1.10.
This seems related to the discussion in Rolling updates should fail early if a pod fails to start #18568

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9
.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-gke.3", GitCommit:"6265b9797fc8680c8395abeab12c1e3bad14069a", GitTreeState:"clean", BuildDate:"2018-07-19T23:02:51Z", GoVersi
on:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Google Kubernetes Engine
OS (e.g. from /etc/os-release): COS

cc @joe-boyce

The text was updated successfully, but these errors were encountered:

MrTrustor · 2018-08-10T10:23:52Z

/sig apps
/sig scheduling

joe-boyce · 2018-08-20T11:30:16Z

Anybody have any ideas on this one?

enisoc · 2018-08-21T17:24:05Z

As far as I can tell, StatefulSet doesn't make any attempt to support this use case, namely using a rolling update to fix a StatefulSet that's in a broken state. If any of the existing Pods are broken, it appears that StatefulSet bails out before even reaching the rolling update code:

kubernetes/pkg/controller/statefulset/stateful_set_control.go

Lines 428 to 435 in 30e4f52

    
           if !isRunningAndReady(replicas[i]) && monotonic { 
        
           	glog.V(4).Infof( 
        
           		"StatefulSet %s/%s is waiting for Pod %s to be Running and Ready", 
        
           		set.Namespace, 
        
           		set.Name, 
        
           		replicas[i].Name) 
        
           	return &status, nil 
        
           }

I haven't found any mention of this limitation in the docs, but it's possible that it was a choice made intentionally to err on the side of caution (stop and make the human decide) since stateful data is at stake and stateful Pods often have dependencies on each other (e.g. they may form a cluster/quorum).

With that said, I agree it would be ideal if StatefulSet supported this, at least for clear cases like this one where deleting a Pod that's stuck Pending is unlikely to cause any additional damage.

cc @kow3ns

mattmb · 2018-09-07T13:34:54Z

+1, I just discovered this and had assumed that it would work more like the Deployment controller.

In https://github.com/yelp/paasta we are programmatically creating/updating Deployments and StatefulSets. For that to make sense I really want them to always attempt to converge to the definition.

This option gives us the option to workaround current StatefulSet limitations around updates See: kubernetes/kubernetes#67250 By default it is false.

fejta-bot · 2018-12-06T14:01:45Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2019-01-05T14:47:24Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2019-02-04T15:31:39Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2019-02-04T15:31:47Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mattmb · 2019-02-07T14:21:06Z

/reopen

k8s-ci-robot · 2019-02-07T14:21:13Z

@mattmb: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mattmb · 2019-02-07T14:21:35Z

Heh, well it was worth a go I suppose...

MrTrustor · 2019-02-08T09:22:14Z

/reopen

k8s-ci-robot · 2019-02-08T09:22:21Z

@MrTrustor: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

huzhengchuan · 2019-03-06T07:49:56Z

+1 meet the same problem
I think need to rollback success. but now it blocked

dave-tock · 2019-03-11T01:00:44Z

+1. This is a pretty big landmine in using StatefulSet, if you ever make any mistake you're stuck with just destroying your StatefulSet and starting over. IOW, if you ever make a mistake with StatefulSet, you need to cause an outage to recover :(

krmayankk · 2019-03-11T17:29:36Z

/remove-lifecycle rotten

with rollingUpdateStrategy: RollingUpdate, operator doens't perform statefulset update it delegates it to the kubernetes statefulset controller. Default strategy is OnDelete, which controlls rolling update process by operator RollingUpdate strategy may be useful, when user wants to perform kubectl restart command or for some reason don't trust rolling update process for operator Fixes annotations merge for pod templates, which make possible kubectl restart command for other services related kubernetes issue with statefulset broken state and why manual rolling update may be needed. kubernetes/kubernetes#67250 #389

saumeya · 2022-04-04T13:46:06Z

This issue also affects our operator - https://github.com/redhat-developer/gitops-operator. Any updates on this problem?

yifan-gu-anchorage · 2022-04-26T02:42:17Z

Hitting the same issue in k8s version 1.21.6

a-hilaly · 2022-05-13T14:55:40Z

/assign

a-hilaly · 2022-05-25T14:44:04Z

/unassign

tomasz-dudziak · 2022-06-13T09:49:56Z

+1

kerthcet · 2022-06-23T03:41:24Z

/assign
I'll take a look as if this is highly needed by the community.

kerthcet · 2022-09-29T02:11:11Z

FYI: KEP initialized here kubernetes/enhancements#3562, glad to hear any advices.

This change ensures that the generate-tenant-config script when running via the zuul-scheduler pod's init-container won't fail when the config location is unaccurate. Having the init-container to success even when the main.yaml file cannot be fully computed ensures that zuul-scheduler pod is in the running state. Indeed, without that change it is impossible (except manually) to recover after a mistake in the config location setting. Three components must restart when the location change: git-server, nodepool, zuul-scheduler. While components restart work well for git-server and nodepool it is broken for zuul-scheduler. According to https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#forced-rollback a rollout will only happen when the Pod is Ready which is not the case when the zuul-scheduler pod failed to start because of the init-container failure. More insights here: kubernetes/kubernetes#67250 The approach implemented in this patch is quite simple as it is just about to bypass the init-container failures and ensures the pod start. The idea is to make it easy to just recover from a wrong config location setting. Change-Id: I6885605d30b5ffc8c4c327dc4590b56316bd7955

Changes to a StatefulSet are not propagated to pods in a broken state (e.g. CrashLoopBackOff) See kubernetes/kubernetes#67250 This is a workaround for the above issue. Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>

wingyiu · 2024-03-26T06:14:18Z

/reopen

k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. kind/bug Categorizes issue or PR as related to a bug. labels Aug 10, 2018

k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 10, 2018

k8s-ci-robot added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Aug 21, 2018

bonifaido mentioned this issue Sep 13, 2018

Support StatefuleSet upgrade with workaround bank-vaults/bank-vaults#157

Merged

matyix pushed a commit to bank-vaults/bank-vaults that referenced this issue Sep 13, 2018

Support StatefuleSet upgrade with workaround

87d6ff6

This option gives us the option to workaround current StatefulSet limitations around updates See: kubernetes/kubernetes#67250 By default it is false.

matyix mentioned this issue Sep 13, 2018

upgrading version of vault in existing cluster not behaving as expected bank-vaults/bank-vaults#154

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 6, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 5, 2019

k8s-ci-robot closed this as completed Feb 4, 2019

k8s-ci-robot reopened this Feb 8, 2019

f41gh7 mentioned this issue Dec 24, 2021

allows to changes rolling update strategy for statefulset applications VictoriaMetrics/operator#402

Merged

saumeya mentioned this issue Apr 8, 2022

fix: delete broken pods of statefulSets argoproj-labs/argocd-operator#604

Merged

2 tasks

ialidzhikov mentioned this issue Apr 22, 2022

pod with CreateContainerConfigError not restarted after stateful set update #109597

Closed

k8s-ci-robot assigned a-hilaly May 13, 2022

k8s-ci-robot unassigned a-hilaly May 25, 2022

k8s-ci-robot assigned kerthcet Jun 23, 2022

kerthcet mentioned this issue Jun 23, 2022

Modifying podAntiAffinity has no effect #110718

Closed

UkonnRa mentioned this issue Aug 24, 2022

Expose .spec.podManagementPolicy to make the pod rollback-able from the broken state jenkinsci/helm-charts#688

Open

kerthcet mentioned this issue Sep 23, 2022

Add EnforcedRollingUpdate to statefulset rolling update strategies kubernetes/enhancements#3541

Open

4 tasks

krmayankk mentioned this issue Oct 6, 2022

Add enforcedRollingUpdate strategy to statefulSet kubernetes/enhancements#3562

Open

hoyhbx mentioned this issue Apr 7, 2023

Zookeeper operator is unable to recover the broken zk statefulset due to an issue in statefulSet controller pravega/zookeeper-operator#547

Open

burmanm mentioned this issue Apr 18, 2023

many properties in CR are fragile, easy to result in broken cluster k8ssandra/cass-operator#518

Open

pwschuurman mentioned this issue Sep 1, 2023

Pending pod are not recreated after updating statefulset resources #120123

Closed

andreasgerstmayr mentioned this issue Sep 11, 2023

Fix ingester StatefulSet reconciliation if ingester is in an unhealthy state grafana/tempo-operator#597

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

StatefulSet - can't rollback from a broken state #67250

StatefulSet - can't rollback from a broken state #67250

MrTrustor commented Aug 10, 2018 •

edited

MrTrustor commented Aug 10, 2018 •

edited

joe-boyce commented Aug 20, 2018

enisoc commented Aug 21, 2018

mattmb commented Sep 7, 2018

fejta-bot commented Dec 6, 2018

fejta-bot commented Jan 5, 2019

fejta-bot commented Feb 4, 2019

k8s-ci-robot commented Feb 4, 2019

mattmb commented Feb 7, 2019

k8s-ci-robot commented Feb 7, 2019

mattmb commented Feb 7, 2019

MrTrustor commented Feb 8, 2019

k8s-ci-robot commented Feb 8, 2019

huzhengchuan commented Mar 6, 2019

dave-tock commented Mar 11, 2019

krmayankk commented Mar 11, 2019

saumeya commented Apr 4, 2022

yifan-gu-anchorage commented Apr 26, 2022

a-hilaly commented May 13, 2022

a-hilaly commented May 25, 2022

tomasz-dudziak commented Jun 13, 2022

kerthcet commented Jun 23, 2022

kerthcet commented Sep 29, 2022

wingyiu commented Mar 26, 2024

StatefulSet - can't rollback from a broken state #67250

StatefulSet - can't rollback from a broken state #67250

Comments

MrTrustor commented Aug 10, 2018 • edited

MrTrustor commented Aug 10, 2018 • edited

joe-boyce commented Aug 20, 2018

enisoc commented Aug 21, 2018

mattmb commented Sep 7, 2018

fejta-bot commented Dec 6, 2018

fejta-bot commented Jan 5, 2019

fejta-bot commented Feb 4, 2019

k8s-ci-robot commented Feb 4, 2019

mattmb commented Feb 7, 2019

k8s-ci-robot commented Feb 7, 2019

mattmb commented Feb 7, 2019

MrTrustor commented Feb 8, 2019

k8s-ci-robot commented Feb 8, 2019

huzhengchuan commented Mar 6, 2019

dave-tock commented Mar 11, 2019

krmayankk commented Mar 11, 2019

saumeya commented Apr 4, 2022

yifan-gu-anchorage commented Apr 26, 2022

a-hilaly commented May 13, 2022

a-hilaly commented May 25, 2022

tomasz-dudziak commented Jun 13, 2022

kerthcet commented Jun 23, 2022

kerthcet commented Sep 29, 2022

wingyiu commented Mar 26, 2024

MrTrustor commented Aug 10, 2018 •

edited

MrTrustor commented Aug 10, 2018 •

edited