Skip to content

How to create alert when using loki? #1422

Closed
@orange888

Description

@orange888

How to create alert when using with loki?

Activity

changed the title [-]How to create alert when using with loki?[/-] [+]How to create alert when using loki?[/+] on Dec 16, 2019
pstibrany

pstibrany commented on Dec 16, 2019

@pstibrany
Contributor

Do you mean Alert in Grafana based on log entries? I don't think that is currently possible. /cc @davkal

orange888

orange888 commented on Dec 16, 2019

@orange888
Author

Do you mean Alert in Grafana based on log entries? I don't think that is currently possible. /cc @davkal

Yes, custom alert based on log (loki)?
Can loki subscribe to kafka? kafka -> promtail -> loki? Or kafka -> loki?

pstibrany

pstibrany commented on Dec 16, 2019

@pstibrany
Contributor

Yes, custom alert based on log (loki)?

David will give you a definitive answer, but I don't think it's currently supported by Grafana (and not sure what kind of support it would need from Loki. Tailing everything all the time doesn't sound like good idea).

Can loki subscribe to kafka? kafka -> promtail -> loki? Or kafka -> loki?

Not at the moment.

davkal

davkal commented on Dec 16, 2019

@davkal
Contributor

Grafana alerting is only available for certain datasources. Even though Loki is not supported yet, Prometheus is supported. And Loki has a prometheus-compatible API, so you can add your Loki as a Prometheus datasource, see #1222. Then you need to set up logql queries that return timeseries and alert on those, e.g., count_over_time({job="foo"} |= "error" [5m]) > 0.
Native alerting w/o the prometheus workaround is still under development.

miklezzzz

miklezzzz commented on Dec 16, 2019

@miklezzzz

@davkal could u please elaborate it in more detail? Where should i set up logql queries? thx

davkal

davkal commented on Dec 16, 2019

@davkal
Contributor

Where should i set up logql queries?

You create a graph panel using the prometheus datasource that points to loki. In the query field you write the logql query and you should see a line graph for your log volume (depends on the query). Then you can add a panel alert.

AnandPalani92

AnandPalani92 commented on Dec 28, 2019

@AnandPalani92

@davkal. - When i create prometheus datasource and that datasources pointed to loki service ,
Facing 404 not found error, see the grafana error logs
t=2019-12-28T00:58:25+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/11/api/v1/query status=404 remote_addr=106.198.13.89 time_ms=202 size=19 referer=https://monitoring-dev.np.example.com/datasources/edit/11/

eraac

eraac commented on Jan 4, 2020

@eraac
Contributor

@anandecemptc URL should be http(s)://<loki-domain>/loki (ex. https://loki.domain.tld/loki). The same as loki datasource but suffix with /loki (and select prometheus datasource)

AnandPalani92

AnandPalani92 commented on Jan 6, 2020

@AnandPalani92

Thanks Eraac.

I have created a prmetheus datasource using loki url (ex. https://loki.domain.tld/loki), during the creation and saving the datasource i checked the datasource and it was worked.

When i checked through the Grafana explore option and metrics are not loaded.

still metric loading...
image

When i searched with some string and its not displayed anything.
image

I created a dashboard and added the query on that dashboard and itsn't loaded anything.
image

cyriltovena

cyriltovena commented on Jan 11, 2020

@cyriltovena
Contributor

You need to use a metric query, see https://github.com/grafana/loki/blob/master/docs/logql.md#counting-logs.

Alternatively I want to add that a safe option is also to add metrics pipeline in promtail and scrape promtail with a Prometheus. This allows to create custom metrics in promtail.

See metrics stage : https://github.com/grafana/loki/blob/master/docs/clients/promtail/pipelines.md

stale

stale commented on Feb 10, 2020

@stale

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

added
staleA stale issue or PR that will automatically be closed.
on Feb 10, 2020
chrono2002

chrono2002 commented on Apr 28, 2020

@chrono2002

Doesn't work for now. Alerter fires:

Object
firing:true
state:""
conditionEvals:" = true"
timeMs:"1.050ms"
error:"tsdb.HandleRequest() error bad_response: readObjectStart: expect { or n, but found p, error found in #1 byte of ...|parse error|..., bigger context ...|parse error : syntax error: unexpected $end |..."
logs:Array[1]
0:Object
message:"Condition[0]: Query"
data:Object

19 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleA stale issue or PR that will automatically be closed.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @davkal@pstibrany@cyriltovena@eraac@DylanGuedes

        Issue actions

          How to create alert when using loki? · Issue #1422 · grafana/loki