Open
Description
/kind bug
What happened:
I updated a StatefulSet with a non-existent Docker image. As expected, a pod of the statefulset is destroyed and can't be recreated (ErrImagePull). However, when I change back the StatefulSet with an existing image, the StatefulSet doesn't try to remove the broken pod to replace it by a good one. It keeps trying to pull the non-existing image.
You have to delete the broken pod manually to unblock the situation.
Related Stackoverflow question
What you expected to happen:
When rolling back the bad config, I expected the StatefulSet to remove the broken pod and replace it by a good one.
How to reproduce it (as minimally and precisely as possible):
- Deploy the following StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "standard"
resources:
requests:
storage: 10Gi
- Once the 3 pods are running, update the StatefulSet spec and change the image to
k8s.gcr.io/nginx-slim:foobar - Observe the new pod failing to pull the image.
- Roll back the change.
- Observe the broken pod not being deleted.
Anything else we need to know?:
- I observed this behaviour both on 1.8 and 1.10.
- This seems related to the discussion in Rolling updates should fail early if a pod fails to start #18568
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9
.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-gke.3", GitCommit:"6265b9797fc8680c8395abeab12c1e3bad14069a", GitTreeState:"clean", BuildDate:"2018-07-19T23:02:51Z", GoVersi
on:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration: Google Kubernetes Engine
- OS (e.g. from /etc/os-release): COS
cc @joe-boyce
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Needs Triage
Status
Needs Triage
Milestone
Relationships
Development
Participants
Activity
MrTrustor commentedon Aug 10, 2018
/sig apps
/sig scheduling
joe-boyce commentedon Aug 20, 2018
Anybody have any ideas on this one?
enisoc commentedon Aug 21, 2018
As far as I can tell, StatefulSet doesn't make any attempt to support this use case, namely using a rolling update to fix a StatefulSet that's in a broken state. If any of the existing Pods are broken, it appears that StatefulSet bails out before even reaching the rolling update code:
kubernetes/pkg/controller/statefulset/stateful_set_control.go
Lines 428 to 435 in 30e4f52
I haven't found any mention of this limitation in the docs, but it's possible that it was a choice made intentionally to err on the side of caution (stop and make the human decide) since stateful data is at stake and stateful Pods often have dependencies on each other (e.g. they may form a cluster/quorum).
With that said, I agree it would be ideal if StatefulSet supported this, at least for clear cases like this one where deleting a Pod that's stuck Pending is unlikely to cause any additional damage.
cc @kow3ns
mattmb commentedon Sep 7, 2018
+1, I just discovered this and had assumed that it would work more like the Deployment controller.
In https://github.com/yelp/paasta we are programmatically creating/updating Deployments and StatefulSets. For that to make sense I really want them to always attempt to converge to the definition.
Support StatefuleSet upgrade with workaround
Support StatefuleSet upgrade with workaround
fejta-bot commentedon Dec 6, 2018
Issues go stale after 90d of inactivity.
Mark the issue as fresh with
/remove-lifecycle stale.Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with
/close.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
138 remaining items