/sig node

Evicted pods in 1.7.5 are a madness, the deletion of those pods are delayed by days!!!!, for example, I have a pod since 17 days was evicted an appear in the pod list:

mynamespace nginxrepo-2549817493-0p91t 0/1 Evicted 0 17d
mynamespace linting-umbrellabird-monocular-api 0/1 Evicted 0 17d

In the case of nginxrepo, the deployment does not exist anymore, but the pods are present in the list of pods as evicted!!!

Also delete pods that does not match node selector criteria:

nfs-3970785943-5rnn7 0/2 MatchNodeSelector 0 17d

After 17 days the pod appear in the list!!. This behavior affect for example in Grafana, because the pods appear in the list of available pods for monitoring, and of course, are evicted!!.

By the way @rfranzke this is not a feature request, this is an issue!!!! Please, could you re-tag the case?

Regards

/kind bug

/remove-kind feature

Thank you @rfranzke

is this a duplicate of #54525

from #54525 (comment) it sounds like this is intentional, though I'm not sure what is expected to clean up pods in this case

though I'm not sure what is expected to clean up pods in this case

The PodGCController in the controller manager?

A quick workaround we use, is to delete all evicted pods manually after an incident:
kubectl get pods --all-namespaces -ojson | jq -r '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | .metadata.name + " " + .metadata.namespace' | xargs -n2 -l bash -c 'kubectl delete pods $0 --namespace=$1'
Not as nice as automatic delete, but it works. (Tested with 1.6.7, i heard in 1.7 you need to add --show-all)

thank you @krallistic, I apply your workaround as cronjob a long time ago but is not a right way!
@liggitt is not duplicated, this is not the case when timestamp are not set properly for differents deployments (daemonset, statefulset, etc), here, appear that all of deployments does not have timestamp for deletion; and this affect the monitoring of the kubernetes environment!!

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

I suppose, this issue can be closed, because the evicted pods deletion can be controlled through settings in kube-controller-manager.

For those k8s users who hit the kube-apiserver or etcd performance issues due to too many evicted pods, i would recommend updating the kube-controller-manager config to set --terminated-pod-gc-threshold 100 or similar small value. The default GC threshold is 12500, which is too high for most etcd installations. Reading 12500 pod records from etcd takes seconds to complete.

Also ask yourself why are there so many evicted pods? Maybe your kube-scheduler keeps scheduling pods on a node which already reports DiskPressure or MemoryPressure? This could be the case if the kube-scheduler is configured with a custom --policy-config-file which has no CheckNodeMemoryPressure or CheckNodeDiskPressure in the list of policy predicates.

$ kube-controller-manager --help 2>&1|grep terminated
      --terminated-pod-gc-threshold int32                                 Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If <= 0, the terminated pod garbage collector is disabled. (default 12500)

Thanks @kabakaev for pointing that out. Didn't know that this can be configured. Let's close the ticket then.

@kabakaev - wouldn't pod gc cover all pods (including terminated pods for other reasons) - what if we just want evicted pods to be cleaned up periodically?

the issue still opened, there no reply for @so0k commented on Feb 27, 2018

@so0k
I created a cron job using a Yaml file with this config (need to fix the formatting, check https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/) :

apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: delete-failed-pods
spec:
schedule: "*/30 * * * *"
failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
containers:
- name: kubectl-runner
image: wernight/kubectl
command: ["sh", "-c", "kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f -"]
restartPolicy: OnFailure

Create the task with kubectl create -f "PATH_TO_cronjob.yaml"

Check the status of the task with kubectl get cronjob delete-failed-pods

Delete the task with delete cronjob delete-failed-pods

Statefulset will auto delete Failed pod

Why does kubernetes keep evicted pod, and what is the purpose of this design？

One guess for the reason: so that you can look at the failed Pods and see what's happening in the cluster more easily (both in the API and in the metrics). If the Pods immediately disappeared, you'd probably need to use logs to discover what's happening, which is arguably more difficult.

When I have too many Evicted pods I use the following command: kubectl get pods -A | grep Evicted | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod --force --grace-period=0