ingestion rate limit exceeded #1923
Description
Describe the bug
I'm getting the following lines in loki when sending logs from promtail (using static_config to scrape logfiles):
level=warn ts=2020-04-09T09:15:05.866134665Z caller=client.go:242 component=client host=172.29.95.195:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): ingestion rate limit (8388608 bytes) exceeded while adding 311 lines for a total size of 102169 bytes"
I don't quiet understand this line or is it misleading?
It says "adding 311 lines for a total size of 102169 bytes". But the total size of 102169 bytes is less than the ingestion limit of 8388608 bytes.
Or does it mean that it tries to store 311 * 102169 = 31.774.559 bytes of data, thus exceeding the ingestion rate limit?
To Reproduce
Steps to reproduce the behavior:
- Started Loki v1.0.4
- Started Promtail v1.0.4
- let promtail parse a huge logfile
Expected behavior
I'd like to understand what this error exactly means.
And also how to avoid it. :)
Environment:
- Infrastructure: bare-metal
Screenshots, Promtail config, or terminal output
Loki limits config:
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
ingestion_rate_mb: 8
Promtail loki-batch-size is set to default.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Participants
Activity
owen-d commentedon Apr 13, 2020
The message is describing which request triggered the rate limiter. Likely this is because previous batches sent from promtail consumed the remainder of the ingestion budget. If you aren't seeing these regularly, it's ok - promtail is designed to handle backoffs and continue ingestion. If these messages are common, increase the
ingestion_rate_mband/or theingestion_burst_sizeconfig :)https://github.com/grafana/loki/tree/master/docs/configuration#limits_config
rndmh3ro commentedon Apr 15, 2020
I had this problem fairly often during my tests and the initial setup if promtail. Probably because it was indexing all these huge system-logfiles at the same time.
However increasing the
ingestion_burst_sizeto 16mb seems to have fixed even these problems.Thanks for your help!
pgassmann commentedon Oct 25, 2021
Current Link to
limits_configdocumentation: https://grafana.com/docs/loki/latest/configuration/#limits_configsmhhoseinee commentedon Feb 2, 2022
I added following config :
again logstash shows the same error and stops sending log to loki:
any solution ?
yakob-aleksandrovich commentedon Feb 3, 2022
TooManyRequestssounds a bit like the issue here: #4613. Try with the following config changesIf I were you, I would bring the ingestion rate and size down to something below the 100. Your current settings allow for 1GB of data per query. That sounds a bit too much...
LinTechSo commentedon Mar 27, 2022
Hi, same issue. I had this problem during my storage migration from filesystem to minio s3 in loki 2.4.2
got "vector_core::stream::driver: Service call failed. error=ServerError { code: 429 } request_id=5020
" error from my log shipper
any updates ?
yakob-aleksandrovich commentedon Mar 28, 2022
Hi @LinTechSo,
I have never seen this specific message, but
code 429hints at the fact that this is the same root cause.Can I assume that you never had this issue when you were running on a local filesystem?
From my own experience, interacting with S3 (AWS or Minio) introduces a lag issue. Writing and reading from local filesystem is very fast, but using S3 on the backend slows things down considerably. This speed issue might affect the 'too many requests' issue, as it will take longer to release connections for re-use.
From your post, I don't know if you are still reading experiencing these issues. If you do, you might want to introduce a 'holdoff' mechanism in your migration script. Basically, give Loki now and then some time to breathe: wait a couple of seconds before sending the next batch of data.
There is also the possibility of setting up multiple Loki instances, so that the load can be spread out. But I have no experience with that.
LinTechSo commentedon Apr 8, 2022
thanks, @yakob-aleksandrovich . would you please explain more what should I do?
how can I tell Loki to wait a couple of seconds before sending the next batch of data?
yakob-aleksandrovich commentedon Apr 8, 2022
Depending on how you feed your data into Loki, this is possible or not.
I'm running a custom Python script, feeding logs into Loki. In this script, I've implemented a sleep(1).
fengxsong commentedon Apr 24, 2022
When this happened to me, I checked the
ingester's logs. It tells that some logs like "Maximum byte rate per second per stream", then I've increased settings forlimits_config. per_stream_rate_limitandlimits_config.per_stream_rate_limit_burst. Problem solved:)rmn-lux commentedon Sep 28, 2022
My experimental config, which seemed to help get rid of the 429 code. Might come in handy.
dellnoantechnp commentedon Dec 28, 2022
promtail inspect logs:
but we have only 6 labels, is [category, filename, namespace, nodename, pod, type].
loki limits_config:
ingester logs:
My total log size less than 150G during 30 days.
Why return "429 Maximum active stream limit exceeded, reduce the number of active streams" ?