You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
David will give you a definitive answer, but I don't think it's currently supported by Grafana (and not sure what kind of support it would need from Loki. Tailing everything all the time doesn't sound like good idea).
Can loki subscribe to kafka? kafka -> promtail -> loki? Or kafka -> loki?
Grafana alerting is only available for certain datasources. Even though Loki is not supported yet, Prometheus is supported. And Loki has a prometheus-compatible API, so you can add your Loki as a Prometheus datasource, see #1222. Then you need to set up logql queries that return timeseries and alert on those, e.g., count_over_time({job="foo"} |= "error" [5m]) > 0.
Native alerting w/o the prometheus workaround is still under development.
pstibrany, 3nin6, laszlocph, SunSparc, vuongxuongminh and 4 more
You create a graph panel using the prometheus datasource that points to loki. In the query field you write the logql query and you should see a line graph for your log volume (depends on the query). Then you can add a panel alert.
miklezzzz, JohnCMcDonough, anishlal, aqt01, laszlocph and 4 moreAnandPalani92
@davkal. - When i create prometheus datasource and that datasources pointed to loki service ,
Facing 404 not found error, see the grafana error logs
t=2019-12-28T00:58:25+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/11/api/v1/query status=404 remote_addr=106.198.13.89 time_ms=202 size=19 referer=https://monitoring-dev.np.example.com/datasources/edit/11/
@anandecemptc URL should be http(s)://<loki-domain>/loki (ex. https://loki.domain.tld/loki). The same as loki datasource but suffix with /loki (and select prometheus datasource)
I have created a prmetheus datasource using loki url (ex. https://loki.domain.tld/loki), during the creation and saving the datasource i checked the datasource and it was worked.
When i checked through the Grafana explore option and metrics are not loaded.
still metric loading...
When i searched with some string and its not displayed anything.
I created a dashboard and added the query on that dashboard and itsn't loaded anything.
Alternatively I want to add that a safe option is also to add metrics pipeline in promtail and scrape promtail with a Prometheus. This allows to create custom metrics in promtail.
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.
Activity
[-]How to create alert when using with loki?[/-][+]How to create alert when using loki?[/+]pstibrany commentedon Dec 16, 2019
Do you mean Alert in Grafana based on log entries? I don't think that is currently possible. /cc @davkal
orange888 commentedon Dec 16, 2019
Yes, custom alert based on log (loki)?
Can loki subscribe to kafka? kafka -> promtail -> loki? Or kafka -> loki?
pstibrany commentedon Dec 16, 2019
David will give you a definitive answer, but I don't think it's currently supported by Grafana (and not sure what kind of support it would need from Loki. Tailing everything all the time doesn't sound like good idea).
Not at the moment.
davkal commentedon Dec 16, 2019
Grafana alerting is only available for certain datasources. Even though Loki is not supported yet, Prometheus is supported. And Loki has a prometheus-compatible API, so you can add your Loki as a Prometheus datasource, see #1222. Then you need to set up logql queries that return timeseries and alert on those, e.g.,
count_over_time({job="foo"} |= "error" [5m]) > 0
.Native alerting w/o the prometheus workaround is still under development.
miklezzzz commentedon Dec 16, 2019
@davkal could u please elaborate it in more detail? Where should i set up logql queries? thx
davkal commentedon Dec 16, 2019
You create a graph panel using the prometheus datasource that points to loki. In the query field you write the logql query and you should see a line graph for your log volume (depends on the query). Then you can add a panel alert.
AnandPalani92 commentedon Dec 28, 2019
@davkal. - When i create prometheus datasource and that datasources pointed to loki service ,
Facing 404 not found error, see the grafana error logs
t=2019-12-28T00:58:25+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/11/api/v1/query status=404 remote_addr=106.198.13.89 time_ms=202 size=19 referer=https://monitoring-dev.np.example.com/datasources/edit/11/
eraac commentedon Jan 4, 2020
@anandecemptc URL should be
http(s)://<loki-domain>/loki
(ex.https://loki.domain.tld/loki
). The same as loki datasource but suffix with/loki
(and select prometheus datasource)AnandPalani92 commentedon Jan 6, 2020
Thanks Eraac.
I have created a prmetheus datasource using loki url (ex. https://loki.domain.tld/loki), during the creation and saving the datasource i checked the datasource and it was worked.
When i checked through the Grafana explore option and metrics are not loaded.
still metric loading...

When i searched with some string and its not displayed anything.

I created a dashboard and added the query on that dashboard and itsn't loaded anything.

cyriltovena commentedon Jan 11, 2020
You need to use a metric query, see https://github.com/grafana/loki/blob/master/docs/logql.md#counting-logs.
Alternatively I want to add that a safe option is also to add metrics pipeline in promtail and scrape promtail with a Prometheus. This allows to create custom metrics in promtail.
See metrics stage : https://github.com/grafana/loki/blob/master/docs/clients/promtail/pipelines.md
stale commentedon Feb 10, 2020
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.
chrono2002 commentedon Apr 28, 2020
Doesn't work for now. Alerter fires:
Object
firing:true
state:""
conditionEvals:" = true"
timeMs:"1.050ms"
error:"tsdb.HandleRequest() error bad_response: readObjectStart: expect { or n, but found p, error found in #1 byte of ...|parse error|..., bigger context ...|parse error : syntax error: unexpected $end |..."
logs:Array[1]
0:Object
message:"Condition[0]: Query"
data:Object
19 remaining items