@kubernetes/sig-node-feature-requests

It is not possible to change OOM behavior currently. Kubernetes (or runtime) could provide your container a signal whenever your container is close to its memory limit. This will be on a best effort basis though because memory spikes might not be handled on time.

FYI using this crutch atm https://github.com/grosser/preoomkiller

any idea what would need to change to make OOM behavior configureable ?

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

/remove-lifecycle stale
/cc @dashpole

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Was this meant to be closed?

It seems like @yujuhong meant to say /remove-lifecycle rotten?

/remove-lifecycle rotten

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

When the node is reaching OOM levels I guess I understand some SIGKILLs happening but when a pod is reaching it's manually set resource limit it also gets a SIGKILL. As the initial post mentions, this can cause a lot of harm.

As a workaround we're going to try and make the pod unhealthy before it reaches the memory limit to get a graceful shutdown.
Kubernetes sending this signal beforehand would solve the issue.

If I want this feature created, how should I go about it? Should I provide a PR with code changes or ping someone to make a proposal?

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

This remains an active issue.

There appears to be no way to gracefully handle OOMKilled at the moment.

@dashpole Can this be re-opened?

/reopen
/remove-lifecycle rotten

/remove-lifecycle stale

for reasons outlined in #40157 (comment) we can't just change the signal delivered. OTOH we can integrate with some OOM daemons, but this would require a separate discussion and KEP.

Kubernetes does not use issues on this repo for support requests. If you have a question on how to use Kubernetes or to debug a specific issue, please visit our forums.

/remove-kind feature
/kind support
/close

@fromanirh: Closing this issue.

@fromanirh this is not a support request, it's a legit feature request.

@fromanirh Can you reopen this, please? It's clearly a feature request, not a support case.

for reasons outlined in #40157 (comment) we can't just change the signal delivered. OTOH we can integrate with some OOM daemons, but this would require a separate discussion and KEP.

it is a feature request for the Kernel not for Kubernetes , the kernel generates the SIGKILL

@aojea there non-kernel solutions here, such as triggering graceful shutdowns at a threshold memory usage (e.g. 95%) before the hard limit is reached.

@aojea there non-kernel solutions here, such as triggering graceful shutdowns at a threshold memory usage (e.g. 95%) before the hard limit is reached.

oh, that is not clear from the title and from the comments, sorry

for reasons outlined in #40157 (comment) we can't just change the signal delivered. OTOH we can integrate with some OOM daemons, but this would require a separate discussion and KEP.

so it should be retitled or open a new issue with the clear request ... and for sure that will need a KEP

Sure, how about titling it: "Add graceful memory usage-based SIGTERM before hard OOM kill happens"

Is there another Issue that's been recreated for this? I can't find one in the Issues list. If not, I can create a new feature request issue.

So an issue has been created already ? if so, would be nice to reference the link here.

@ffromani this is not a support request, this is a feature request that has been bumped for 6 years now, and i just found a usecase for, just as countless users can find usecases for here

Please reopen the issue

Adding my own voice -- a service being unable to shutdown gracefully can cause all kinds of harm, and causes harm for my organization as well. I thought kube already sent a SIGTERM first and was trying to substantiate that. Is this feature request still un-implemented? When memory limit is reached, pods are SIGKILLed immediately with no chance to shutdown gracefully?

/reopen
/kind feature

there's obvious interest in this feature, but would surely need a KEP and someone sheparding the feature

@ffromani: Reopened this issue.

/remove-kind support
/triage accepted

FWIW my user story is:

I have a Ruby application (developed not by us) that has a slight memory leak over many days, it will balloon to a certain amount of memory, and then gradually keep adding memory.

For this, I started using oomhero to attempt to manage, but it does not seem to be perfect, nor always work. I want to set a hard limit on certain containers, and have applications get told a SIGTERM just before or after they reach that limit, so it can shut down normally and clean up.

Sometimes, these containers use imagemagick or ffmpeg to process some media, and these can balloon memory usage. The container can shut itself down and kick the job back to the queue when this happens, in a way that does not leave resident keepalive queues (sidekiq) and other things that need to be cleaned out every so often.

Implementation ideas: The container that has been told a SIGTERM for overriding the memory limit gets put first as a candidate for the OOMkiller, this could mean that, even if the application ignores the SIGTERM (or requires more resources for shutting down), it can overprovision the remaining system resources before being killed itself.

This could be a toggle per-container, from "hard" OOM kill to a "hard" OOM kill (default).