Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature request: Add agent support for beyond WAL persistent storage and metric export #9607

Open
bwplotka opened this issue Oct 28, 2021 · 11 comments

Comments

@bwplotka
Copy link
Member

Just an idea.

Currently, we are creating an implementation that has a scrape and WAL only https://github.com/prometheus/prometheus/pull/8785/files

I believe we can go further and offer persistent storage beyond WAL, so if agent is disconnected for longer than 2h it would simply compact block and upload to object storage. This is would an amazing solution for "Too old data" ingestion problem pointed in cortexproject/cortex#2366

Would work well for both Cortex and Thanos, might be more tricky for other vendors since this escapes just Remote Write.

Something to consider 🤗

cc @tomwilkie @rfratto @gouthamve

@rfratto
Copy link
Member

rfratto commented Oct 28, 2021

Does this require direct access to the object stores used by Cortex/Thanos? (ie no support for a hosted Prometheus backed by either?)

@bwplotka
Copy link
Member Author

Either that or Thanos/Cortex could provide API for upload - whatever works 🤷🏽

@roidelapluie
Copy link
Member

Creating blocks, why not, but uploading to bucket storage changes the scope of the project and should be discussed on the dev mailing list or during dev summits.

@codesome
Copy link
Member

codesome commented Nov 1, 2021

I believe we can go further and offer persistent storage beyond WAL, so if agent is disconnected for longer than 2h it would simply compact block and upload to object storage.

This idea seems applicable to current Prometheus too where the compaction is already done, just short of shipping the block. It will be easiest (and fastest) to try it out there first if we ever decide to allow it.

@bwplotka
Copy link
Member Author

bwplotka commented Nov 1, 2021

Totally to be discussed on next Dev Summit 👍🏽 Happy to shepherd this discussion.

@dginhoux
Copy link

Hi,

That's a great idea.
So, it scrape targets and forward to a tsdb backend like victoriametrics... very simple and very usefull mode.

How, in agent mode, will work alerting with altermanager ?

I've see the log "field alerting is not allowed in agent mode"... i made a mistake or i don't understand something.

@roidelapluie
Copy link
Member

In Agent mode, the alerting has currently to be done by the remote write target (in your case, victoriametrics).

@valyala
Copy link

valyala commented Dec 9, 2021

@dginhoux , if you forward data from Prometheus agent to VictoriaMetrics, then alerting rules can be evaluated at VictoriaMetrics side with vmalert.

@roidelapluie
Copy link
Member

@bwplotka did we end up discussing this?

@carlosrmendes
Copy link

any news about this feature?

@fede843
Copy link

fede843 commented Oct 3, 2023

is this under development?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants