My understanding is that it is used to stop rollout in case of any problems in the ring and make operator to investigate and fix (or just "forget" bad entry).

@tomwilkie will surely know reasoning here.

I also believe it is to halt a rolling update.

Related: #1521

I recently add #2936 to help users diagnose this problem, doesn't address the question of changing the behavior but should make it easier for operators to diagnose and recover

I recently add #2936 to help users diagnose this problem, doesn't address the question of changing the behavior but should make it easier for operators to diagnose and recover

Yes, it's a starting point. However, I think the main use case for people is having Cortex self-healing. If you run on Kubernetes deployments (so so each new ingester will have a random ID), if you just loose 1 ingester its pod will be rescheduled by Kubernetes but will not join the ring because the previous one (that didn't cleanly shutdown) is unhealthy within the ring.

will not join the ring

I run ingesters in a Deployment and I don't think I've observed that.

will not join the ring

I run ingesters in a Deployment and I don't think I've observed that.

They will join the ring, but the healthcheck for pod readiness senses that the ring is unhealthy and will lead to the pod that joined to replace the failed one to restart as well.

Not seen that either.
Can you show log files where a pod decides to restart because it is not ready?

Today I see a message was added as part of the gossiping changes:

level=warn ts=2020-09-09T22:14:14.839462614Z caller=lifecycler.go:230 msg="found an existing instance(s) with a problem in the ring, this instance cannot complete joining and become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance ingester-5c45496986-k74wz past heartbeat timeout"

which is lying because the ingester printing this message is ACTIVE in the ring (although it is not ready).

Today I see a message was added as part of the gossiping changes:

It was actually added in #2936, because this state is confusing for people.

It is perhaps not very precise, but what it's trying to say is that Ingester cannot report "ready" via /ready handler, because there are other ingesters in the ring that are not ACTIVE.

To clarify my complaint:
"this instance cannot become ready until this problem is resolved" would be ok.
"this instance cannot complete joining" is incorrect: the ingester has joined the ring and is in ACTIVE state.

@bboreham I created #3158 to correct the message.

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

You can use the ring http endpoint to "forget" unhealthy entries and not have to scale to zero and back up.

Both Cortex and Loki both have a configuration parameter now to "auto forget" unhealthy ring entries too

Both Cortex and Loki both have a configuration parameter now to "auto forget" unhealthy ring entries too

I believe Cortex has this for store-gateway, ruler and alertmanager. Not ingester.

if we just scale 3 pods up from 0 it cannot handle the situation

@zetaab your logs say there were two pods not heartbeating. No reason to connect that symptom with scaling.

"instance cortex-ingester-7444b9cb6d-lczgp past heartbeat timeout"

We also have this problem where the ingester set doesn't come back up because of this catch-22 where the first ingester doesn't report readiness since it can't contact the others in the set, and the others are not scheduled to start until the first instance reports readiness.

I think this is the same issue reported here, and the issue is closed. Is that because there is a fix, or is the manual removal of instances from the ring deemed sufficient or something else? Just trying to get a grip on whether it needs to be pushed further.

Seeing same issue on v2.5.0. Any help/pointers to get around this? Its biting us hard in prod :(

Having exactly the same issue with 2.6.1. A single ingester got OOMKilled and brought the entire ingestion pipeline down. I couldn't even get a size=1 ingester ring to come up... it just flaps between PENDING, ACTIVE, and LEAVING, the same as a much larger one. To say the least, it is not building confidence in the ability to keep this thing stable enough to use in prod... there aren't even any docs for what to do about this.

maybe this will help

ingester:
      autoforget_unhealthy: true

Seeing the same issue on v1.13.1; to be honest i'm not sure when /how exactly the ingestor pods "died" but let's assume OOMKilled, the thing is; the standard recovery pattern by K8S doesn't work since the new ingester's don't report ready since the old instances are still in the ring.

@rmn-lux this helped thanks

facing the same issue, is it resolved?

facing the same issue, is it resolved?

still facing this issue

level=warn ts=2024-06-23T15:14:04.57261201Z caller=lifecycler.go:291 component=ingester msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 192.168.116.137:9095 past heartbeat timeout"

still facing this issue

level=warn ts=2024-06-23T15:14:04.57261201Z caller=lifecycler.go:291 component=ingester msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 192.168.116.137:9095 past heartbeat timeout"

There's a flag that might make this better for you:

 # When enabled the readiness probe succeeds only after all instances are
  # ACTIVE and healthy in the ring, otherwise only the instance itself is
  # checked. This option should be disabled if in your cluster multiple
  # instances can be rolled out simultaneously, otherwise rolling updates may be
  # slowed down.
  # CLI flag: -ingester.readiness-check-ring-health
  [readiness_check_ring_health: <boolean> | default = true]

You can try setting it to false.

Also, disabling the heartbeat on the ingester may be useful, because the distributor has its own health check to the ingester, so it's kind of redundant.

I have the same issue when enabling pattern_ingestor, same solution as ingestor?