I have noted that the issue does not seem to occur when periodSeconds is not specified.

If I create the above deployment without the periodSeconds key/value for readinessProbe specified, timeing the kubectl rollout status deploy/nginx immediately after kubectl create -f deployment.yaml results in timings that seem more fitting/expected:

real	0m18.177s
user	0m0.147s
sys	0m0.021s

real	0m17.621s
user	0m0.134s
sys	0m0.022s

real	0m16.340s
user	0m0.154s
sys	0m0.021s

I am also experiencing the same issue, and have been able to successfully reproduce it with @RochesterinNYC's minimal deployment manifest. With periodSeconds set to 300, my replicaset took over a minute to reach 1/1, yet without periodSeconds specified, my replicaset reached 1/1 in under 5 seconds.

/sig node

/assign

Seems likely that this is caused by the jitter/sleep that kubelet does before sending the first probe:

Noting also that the impetus for this issue is the described/intended behavior of readiness probes/liveness probes as dictated at https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

Probes have a number of fields that you can use to more precisely control the behavior of liveness and readiness checks:

- initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated.
- periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.
- timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1.

If there is something else that should be happening or the behavior observed in this issue is actually "expected" when these options are configured together, then that should potentially be called out in the documentation.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot: Closing this issue.

I don't think this should have been left to go stale because this issue is real and goes against how the probes are described in the docs. We have a probe that, due to API limits, can only happen every 600s, but that should not mean the first probe takes up-to 10m to happen after a pod first starts. I understand the logic about not wanting all probes to happen at once if the kubelet crashes, but in the case of a single pod creation it should not be a problem.

/reopen
/remove-lifecycle rotten

I think this should be re-opened as I was about to create a new issue with essentially the same problem.

I was expecting readiness probes (and presumably liveness probes as well) to execute the first probe immediately upon initialDelaySeconds expiring. Instead, it may take a substantial amount of time before the pod is marked "Ready" upon the expiration of the initial wait. The suggestion about the jitter calculation above looks promising as my results were very confusing from my experimentation with shorter/longer values for periodSeconds. The more I tested this the more confused I got. Using a simple ls command, initial delay of 30s, and period of 200s, my pod entered the ready state at approximately 2m30s after the container start event. Conversely, an initial delay of 0 with a wait of 3600 essentially never becomes ready, despite the command being guaranteed to succeed.

All of the examples on the Configure Liveness and Readiness Probes page use an initial delay shorter than period seconds, and generally short values across the board. Text blocks like this:

In the configuration file, you can see that the Pod has a single Container. The periodSeconds field specifies that the kubelet should perform a liveness probe every 5 seconds. The initialDelaySeconds field tells the kubelet that it should wait 5 second before performing the first probe.

leave it semi-ambiguous as to whether the probe is executed immediately at that 5 second mark or has some built-in delay (regardless of whether that delay is based on periodSeconds).

@ankrause: You can't reopen an issue/PR unless you authored it or you are a collaborator.

I am also running into this issue, the readinessProbe doesn't start for quite a long time (longer than the initialDelaySeconds) if the periodSeconds is set to a higher value.
How exactly is this issue closed already?

/reopen.
/remove-lifecycle rotten

It seems to me that something among the following would be needed:
a) update documentation to warn users about using an initialDelaySeconds lower than periodSeconds, and that all timers are only honored approximately because of a random timer of up to periodSeconds is being added
b) update kubelet code to use a lower jitter bounded by initialDelaySeconds for the first probe
c) prevent users from using a initialDelaySeconds lower than periodSeconds

/reopen
/remove-lifecycle rotten

TL;DR:
Please could I suggest to Kubernetes that
a) the documentation is addressed and
b) possibly modify the sleep function and constrain the variability by a smaller, hardcoded, amount independent from readinessPeriod (a random portion of 5sec should be enough for anyone, right...?)

or
c) expose the 'variability' in the readiness probe sleep so that, for time sensitive readienss gates the human can set to zero?

Detail:
I've spent this week looking at instances where we sometimes see a statefulset fail to cluster properly on initial deployment in our wider application
• We have a statefulset and 3 instances which deploy one after the other. We use Parallel rather then OrderedReady to remove dependcies on lower numbered instances. We use an init container to 'sleep', the amount of sleep is based on the pod instance name, instance 0 slepps 0 seconds, instance 1 sleeps n1 seconds, onstance 2 sleeps n2 seconds.
• When the statefulset deploys all pods appear in the endpoint list as notReady and at the point where the pod passes its readiness probe it is marked ready and the enters service rotation (more/better precise terminology for ‘service rotation’ later – just go with it!).
• Peer discovery in application pod doesn't actually care if endpoints are ready or not. When application peer discovery occurs it gets the list of endpoints and tries to talk to everything in the list. Whether or not those endpoints are ready or not it'll try and talk to them anyway. However, you can't talk to a peer pod that is notReady because it's not in service rotation yet, even if you ask for it using the pod’s FQDN (even if the application running in the pod was capable/ready at the point a peer wanted to talk).
• When instance 0 deploys there are 2 peers but none will talk to it (they are not in service rotation, and the application pods are not even deployed yet!) so the application attempt to chat to the peers fails so instance 0 forms a cluster on its own.
• When instance 0 deploys there is a period of time before it is 'ready' according to the readiness probe.
• When instance 1 deploys it sees there are 2 peers. Instance 2 is not yet deployed (container not even started yet) but instance 0 pod is running. At this point the pod must be running and ready otherwise the peering will fail, and instance 1 will form a cluster all by itself – which is bad. So we must ensure that the readiness probe passes for instance 0 before instance 1 attempts to peer. I've looked carefully at the application source code (happens to be RabbitMQ in this case) - during the initial peering process, if it attempts to talk to a peer in the peer list and it isn't possible to establish a TCP connection then it fails immediately. No timeouts, no retries. There are lots of reasons why it may not be possible to establish a connection (DNS outage etc etc) but if the peer isn't in service rotation (is not ready) then it'll certainly fail.
• When instance 2 deploys it sees there are 2 peers. By this time it's pretty much certain that instance 0 has entered the ready state and so 2 can peer with 0. It’s unlikely that 1 is ready yet so it’ll peer with instance 0. Sometimes, if instance 1 is ready then it may pick instance 1 rather than 0
End result:
0 and 2 are co-clustered, 1 is in its own cluster - Rabbit cluster is partitioned and will never 'heal' at this point because instance 1 thinks it's doing the right thing and there's nothing to heal and instance 0 and 2 were never aware of 1.
Or
1 and 2 are co-clustered, 0 is in its own cluster - Rabbit cluster is partitioned and will never 'heal' at this point because instance 0 thinks it's doing the right thing and there's nothing to heal and instance 1 and 2 were never trying to cluster with 0.

So - why, despite our careful initContainer delays and careful Readiness probe delay setting, do we see issues with application cluster formation?
In the k8s kubelet source code there is this little bit of code before the first readiness probe is sent after starting a pod.

`// run periodically probes the container.

func (w *worker) run() {
probeTickerPeriod := time.Duration(w.spec.PeriodSeconds) * time.Second

// If kubelet restarted the probes could be started in rapid succession.
// Let the worker wait for a random portion of tickerPeriod before probing.
time.Sleep(time.Duration(rand.Float64() * float64(probeTickerPeriod)))

probeTicker := time.NewTicker(probeTickerPeriod)

`

FYI:- rand.Float64 returns, as a float64, a pseudo-random number in the range [0.0 to 1.0]

You can see it's not just the initial delay - its initial delay plus a random portion of the probe period (30sec in our case)
In our use today can we have a the following initialDelay 35sec (which is a measured amount of time that it takes after starting the application pod before the application should (comfortably) be ready) and a readinessPeriod of 30sec. So the 1st readiness probe is delayed by a minimum of 35sec upto an additional 30sec (potential total of 65sec). but an initContainer sleep of only 45sec so instance 1 is only 45sec behind instance 0 and so will attempt to peer before instance 0 is marked ready.
This easily causes the situation I describe above where instance 1 wants to peer but kubelet hasn't yet sent a readiness probe to instance 0 and so instance 0 is notReady and is not in service rotation and so instance 1 can not talk to instance 0 even though the application in instance 0 it totally capable at this point. This silly (undocumented) k8s 'feature' is the root cause of our problem.

Now we understand this we can set:

1. the readinessIntialDelay big enough to ensure the application starts and so it passes the 1st readiness probe. Interestingly, the 3rd party chart defaults often show a failure of the 1st readiness probe if the 1st readiness probe is sent close to the readinessInitialDelay (which is 10 sec) but not if the 1st readiness probe is sent toward the end of readinessInitialDelay+readinessPeriod (which is 40sec). (i.e; default initialDelay 10sec is too short)
2. ensure we set the gap between each statefulset instance deploying the application pod to at least readinessInitialDelay+readinessPeriod
   which allows us to now set the initContainer sleep big enough to cover over this gap which can be 65 sec in total, (I am testing with 35sec+30sec+5sec (5sec for safety!!)) which gives 70sec. We could also reduce the period to reduce the variability but I suggest we shouldn't as we don’t want to increase load on any of the kubelets.

Apart from actually making the thing work more reliably, these changes don't change time taken the RabbitMQ cluster to become available to clients much, if at all. The cluster is 'up' once the 1st instance is running and marked ready (and the application is actually ready). So the 1st instance will be ready 35 to 65sec after pod deployment. The defaults of the (Bitnami) chart are actually sometimes worse as you often hit a situation where the 1st readiness probe fails and then you have to wait another 30sec before the next readiness probe (10+30+30).

Please could I suggest to Kubernetes that the documentation is addresses and possibly modify the sleep function and constrain the variability by a smaller, hardcoded, amount independent from readinessPeriod (a random portion of 5sec should be enough for anyone, right...?) or expose the variability value so that, for time sensitive readienss gates the human can set to zero?

Clarification about services, DNS records and ready probes:
"Headless" (without a cluster IP) Services are also assigned a DNS A or AAAA record, depending on the IP family of the service, for a name of the form my-svc.my-namespace.svc.cluster-domain.example. Unlike normal Services, this resolves to the set of IPs of the pods selected by the Service. Clients are expected to consume the set or else use standard round-robin selection from the set.

Because A or AAAA records are not created for Pod names, hostname is required for the Pod’s A or AAAA record to be created. A Pod with no hostname but with subdomain will only create the A or AAAA record for the headless service ( default-subdomain.my-namespace.svc.cluster-domain.example ), pointing to the Pod’s IP address. Also, Pod needs to become ready in order to have a record unless publishNotReadyAddresses=True is set on the Service.
We do not use publishNotReadyAddresses in our RabbitMQ services (neither headless or not-headless). I don't think we want to do publishNotReadyAddresses because we want ready to be meaningful. I want to prevent the situation where the RabbitMQ pod is in its preflight/boot/initialisation phase (where the application is running but its doing things like validating env vars and other environment/runtime shaping activities prior to really actually starting with the expected args/configs) and something talks to it in that phase. After that phase, the RabbitMQ application gets stopped and later (a second or so later) gets restarted (which I think is a bitnami behaviour rather than purely RabbitMQ application requirement). My brain hurts thinking about what the resultant cluster would look like if you talked to the RabbitMQ application in the preflight/boot/initialisation stage.

@JamesTGrant: You can't reopen an issue/PR unless you authored it or you are a collaborator.

Same problema here