I see this is still not implemented. This would really be much appreciated and I think its a pretty logical way to think about it:

[readinessProbe] Is it ready? No
-> Okay lets wait a bit and see [still readinessProbe]

Eventually...
[readinessProbe] Is it ready? Yes
-> Okay we can serve traffic and run checks to see if it remains alive [livenessProbe]

Having liveness probe kill off the pod before it's ready is frustrating and counter-intuitive. As mentioned, that should be handled completely in the readiness probe - with a set limit on time to ready / check iterations. As it stands now I have to create some rough estimates about how long I should delay my liveness probe - which is not always an easy constant to track.

@kubernetes/sig-node-feature-requests

The change is easy to implement however what is the correct behavior given what is done today.
If liveness runs after readiness, the delays that are there today would be odd. Like you already stated, it is hard to pick a value but people did.

Understandably, this will cause unexpected behaviour because of the way people have bent their liveness probes around this at the moment.

I suggest an extra variable on livenessProbe perhaps? Something like waitForReadinessProbe.

And maybe a warning somewhere that on future versions (however far down the line) liveness probes will wait by default. Or just stick with the extra waitForReadinessProbe indefinitely... (it just seems like it should be default behaviour)

I totally like this idea, my general understanding of a livenessProbe was that it is only run on ready containers - I was surprised to find out that this is not the case. (Actually found that out the same way as @tanapoln - when running a container with long boot time for the first time.)

I vote for making waitForReadinessProbe the default, as it makes so much more sense.

Copying and pasting my own comment from the PR:
IIUC, liveness was intended to start before readiness in many cases. A process is running doesn't mean it's ready to serve traffic. It won't be restarted (since it passes liveness checks), but will not be listed as endpoints in a service.

Changing the interaction of these two checks will surprise many users.

@yujuhong I can see that and its a fair understanding. But it doesn't change the fact that there is no easy way to ask your liveness probes to wait until your instances are actually ready- hence causing an early shutdown if you didn't offset it enough. This is not a stable solution at all.

In any case, it seems that the ambiguity of it has caused it to mean different things to different people. I can't think of a time I would consider a pod "alive" but not "ready", so that's why the default behaviour seems strange to me.

I guess the default behaviour should remain the same then. And as suggested, if there is an additional variable such as waitForReadinessProbe defined on livenessProbe, then the behaviour aligns with what we are expecting here.

Unless there was some other kind of probe that acts as a precursor to both the readinessProbe and livenessProbe probes. But that would start to feel excessive I think, for such a seemingly easy concept.

Would it be possible to use a toleration/taint approach for the pod to manage the "state" of this lifecycle in a more persistent way and alleviate the use case of the Kubelet start? I do understand how this could get a bit tricky, where we may need to also then keep the timestamp of the pods initial Readiness check. If the pod gets evicted and replaced or restarted, what should happen then, etc.

In my opinion, as the StatefulSet paradigm continues to increase, this will become an increasingly prevalent issue. In our case we are attempting to run a few things like Kafka and Elasticsearch which, in an exception case can take dramatically longer to start up (rebuilding indices and rebalancing) than a clean restart would do.

We are currently evaluating a few options but having the ability to indicate the Liveness check's initial delay should only start after the Readiness passes the first time would take a lot of the guess work out of it for us.

hmm what about the scenario where you want this as in if the startup configuration is dynamic waiting for a service that doesn't exist anymore. Then you want to restart the configuration step again to pick up the new changes else it will never boot up

Just stumbled upon this problem on rolling out the deployment resource which has a long-ish start-up time. Deployment has both readiness and liveness probes specified. Because the app takes a lot of time to start the failing liveness probe eventually restarts the pod and thus deployment enters the crash/restart loop. This is quite non-intuitive, especially because I use kubectl rollout status deployment ... to wait for rollout to finish which seems to depend only on readiness probe.

IMO, at least on deployment rollout, it would be more logical to start probing for liveness after deployment's pod is ready.

I ran into this same issue when first learning K8s. I agree with the previous comments that logically liveness needs to follow readiness.

How about adding "start before delay if ready" instead of "start when ready" property? so liveness probe would start either after ready status or after it has reached its delay.

It makes sense that K8s has treated these separately so that the correspondence between check and its effect is clear.
Using readiness check means you want to automatically change load in response to check.
Using liveness check means you want to automatically restart (depending on policy) in response to check.

I think we should add an extra parameter for the liveness probe waitForReadinessProbe to make it wait until readiness has succeeded, but that flag should have a default false so it behaves as before, otherwise there may be cases of containers deadlocking on startup (not passed readiness yet but become dead) that are no longer restarted.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

No updates on this? Adding the extra parameter would make a lot of sense. We sometimes do database migrations on pod startup (If something big changed) and that can take an unknown amount of time, so it feels a bit unnecessary to always have the initialDelay very high just to avoid the pod getting killed when doing the migration. If it would wait for readiness instead it would feel much more safe. I understand that this is not always the case, so this extra parameter would solve the issue for all parties I think.

After spending a few hours reading through this issue, all the docs, etc., I'm still finding it annoyingly complex to account for a pod where, the first time it starts up and configures everything, it could take 5-10 minutes, but then after that subsequent pods could take anywhere from < 1 second to 1 minute.

I want the pods to come up quickly, and also stop serving requests quickly if there's a failure in one of the probes, but to account for the fact that initialization the first time could take 10 minutes, I have to leave padding in both my probes (since I can't have one run before the other kicks in) and either make it so no container is 'ready' until after 10 min or so, or containers could be in a bad state (but not marked 'unhealthy') for up to 10 min or so.

I, like others earlier in this thread mentioned, would love to have an annotation or configuration option to have it be like "livenessBeforeReadiness" so I can use that kind of behavior (which seems more intuitive to me?).

For the most part we have been able to solve these kinds of issues with init containers. The init container handles the case where setup takes an unusual amount of time. For the happy path it just exits with a no-op.

@wstrange as has been mentioned, for some of us, init containers aren't the solution because the software we are dealing with can't be modified in a way that would support them. It's great when you can, but it simply isn't possible in all situations. There are of course workaround for that, but as @matthyx has observed, it would be much better if they were unnecessary.

I have to leave padding in both my probes...

Be careful with not executing your probes, as it leaves you blind regarding the state of your container.

The main idea behind something like maxInitialFailureCount is to allow probes to run and fail during initialization, while still recording container states.

@matthyx - That's exactly why I think this issue has had so much uptake. The way liveness/readiness probes currently work is quite fragile and/or nerve-wracking because you can't really optimize for the use case of "pods can take a while to start and be responsive sometimes, and that's okay".

I was looking at #71449 and it looks like there is still a decently-long path towards it hitting a stable release; therefore I have to still find ways to work around this shortcoming for probably the next year or two at least (I'm on EKS... still on 1.11 :P).

I will definitely look into using an initContainer for now, as that feels like it could be the cleanest way to achieve this without artificially inflating the probes' initial timeouts.

The way liveness/readiness probes currently work is quite fragile and/or nerve-wracking because you can't really optimize for the use case of "pods can take a while to start and be responsive sometimes, and that's okay".

I agree, and that's why I have submitted a talk for the next Kubecon to raise awareness and hopefully get smart minds working on an acceptable solution.

I will definitely look into using an initContainer for now, as that feels like it could be the cleanest way to achieve this without artificially inflating the probes' initial timeouts.

You can also try a wrapper for the livenessProbe, as suggested in my previous comment.

Having an init container to handle the expected behavior of a liveness probe to start checking after the readinessprobe seems counter intuitive. An implementation from the past with a configured delay before it starts checking, should not hinder the development of this expected behaviour.

Proposal submitted, wish me luck :-)

Proposal accepted, see you in Barcelona!

In the meantime, could someone merge my KEP (kubernetes/enhancements#860)?

We ended up building a sidecar container that fronts liveness and readiness. However if the main container restarts for some reason we have to reset the haveBeenReadyAtLeastOnce state. We use a postStart hook to call the sidecar container and let it know the current container it's monitoring is new.

This, however, is called on the first boot of the container as well. The side car is written in go so it comes up really quick but still not quick enough because it fails the first postStart hook with a connection refused. This ends up restarting the main container on the first boot a couple of seconds after it's started. So all of our pods now have a single restart.

I'm wondering what the purpose of an httpGet postStart hook is when it's very unlikely that anything will be able to respond to it that quickly. Since it restarts the container it would never be able to come up cause it would keep restarting. Is there something I'm missing here?

Other than having an init container installing it we can't rely on a given binary like curl being present on any container we build/use. Using curl though would allow me to have retries built into the postStart hook. Now, everything starts getting pretty "Rube Goldberg". At this point why don't I just build a go binary that I side-load with an init container to directly call the main process and front all of the retry/wait until readiness logic. Again calling into question the existence of an httpGet postStart hook.

I agree with what a lot of people have already said: this feels like logic that should be abstracted into kubernetes. I don't necessarily know what the solution should be but I think the extra property in the livenessProbe to wait for readiness is a good start.

@from-nibly - FYI, I filed httpGet postStart hook should retry on connection refused/timeout #78286

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

startupProbe is alpha in 1.16!

@matthyx thanks!

From a brief look at the pull request, it looks a bit different to what was talked about here.

Is there any documentation we can look over about how this startupProbe works ?

EDIT: Nevermind, I think this is it here.

@geerlingguy we share the same case. A Laravel App took sometimes up to 4 minutes to start (php-fpm) .
Do you mind to share an example of initContainers that you implemented to overcome the issue of probes ?

@geerlingguy Looks like the new probe startupProbe will solve the issue here : 🎊 🎊 🎊

startupProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 30
  periodSeconds: 10

Reference : https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes

Indeed!