Do you mean Alert in Grafana based on log entries? I don't think that is currently possible. /cc @davkal

Do you mean Alert in Grafana based on log entries? I don't think that is currently possible. /cc @davkal

Yes, custom alert based on log (loki)?
Can loki subscribe to kafka? kafka -> promtail -> loki? Or kafka -> loki?

Yes, custom alert based on log (loki)?

David will give you a definitive answer, but I don't think it's currently supported by Grafana (and not sure what kind of support it would need from Loki. Tailing everything all the time doesn't sound like good idea).

Can loki subscribe to kafka? kafka -> promtail -> loki? Or kafka -> loki?

Not at the moment.

Grafana alerting is only available for certain datasources. Even though Loki is not supported yet, Prometheus is supported. And Loki has a prometheus-compatible API, so you can add your Loki as a Prometheus datasource, see #1222. Then you need to set up logql queries that return timeseries and alert on those, e.g., count_over_time({job="foo"} |= "error" [5m]) > 0.
Native alerting w/o the prometheus workaround is still under development.

@davkal could u please elaborate it in more detail? Where should i set up logql queries? thx

Where should i set up logql queries?

You create a graph panel using the prometheus datasource that points to loki. In the query field you write the logql query and you should see a line graph for your log volume (depends on the query). Then you can add a panel alert.

@davkal. - When i create prometheus datasource and that datasources pointed to loki service ,
Facing 404 not found error, see the grafana error logs
t=2019-12-28T00:58:25+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/11/api/v1/query status=404 remote_addr=106.198.13.89 time_ms=202 size=19 referer=https://monitoring-dev.np.example.com/datasources/edit/11/

@anandecemptc URL should be http(s)://<loki-domain>/loki (ex. https://loki.domain.tld/loki). The same as loki datasource but suffix with /loki (and select prometheus datasource)

Thanks Eraac.

I have created a prmetheus datasource using loki url (ex. https://loki.domain.tld/loki), during the creation and saving the datasource i checked the datasource and it was worked.

When i checked through the Grafana explore option and metrics are not loaded.

still metric loading...

When i searched with some string and its not displayed anything.

I created a dashboard and added the query on that dashboard and itsn't loaded anything.

You need to use a metric query, see https://github.com/grafana/loki/blob/master/docs/logql.md#counting-logs.

Alternatively I want to add that a safe option is also to add metrics pipeline in promtail and scrape promtail with a Prometheus. This allows to create custom metrics in promtail.

See metrics stage : https://github.com/grafana/loki/blob/master/docs/clients/promtail/pipelines.md

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

Doesn't work for now. Alerter fires:

Object
firing:true
state:""
conditionEvals:" = true"
timeMs:"1.050ms"
error:"tsdb.HandleRequest() error bad_response: readObjectStart: expect { or n, but found p, error found in #1 byte of ...|parse error|..., bigger context ...|parse error : syntax error: unexpected $end |..."
logs:Array[1]
0:Object
message:"Condition[0]: Query"
data:Object

I am also seeing this with grafana 6.6.2 and above when trying to create an alert using the prometheus from Loki

firing:true
state:"pending"
conditionEvals:" = true"
timeMs:"2.785ms"
error:"tsdb.HandleRequest() error bad_response: readObjectStart: expect { or n, but found p, error found in #1 byte of ...|parse error|..., bigger context ...|parse error : syntax error: unexpected end|..."

@chrono2002 @houstonj1 I'm working on a fix #2020

btw which Loki are you using ? Cloud / Tanka / Helm ?

@cyriltovena We are deploying Loki v1.4.1 with a custom Helm chart

You shouldn't be impacted then. Hum.

To get things working again, I rolled back to Grafana v6.5.3

Doesn't work for now. Alerter fires:

Upgraded to latest Loki. Everything is ok now.

hello,
error Cannot read property 'name' of undefined

I was able to do it. Alarm is working. Via PrometheusDatasource (LOKI as db). The tricky thing is the job names, which are prompted from loki as like this e.g. : {filename="C:\abc\xyz\Server\log\Server_Log.txt", job="My_Server_logs"}

Use regex to search text from the job name. That's it.
count_over_time({job =~ ".*My_Server_logs.*"} |~ "ERROR" [1m])
Alarms do work from Grafana/Graph panel.

Naturally it would be nice that LOKI datasource would support directly alarms.

Possible with the latest grafana.

Latest version of grafana supports Loki Alert natively

i'm not able to get the Loki alerting working with Grafana now. It used to work in a different implementation 8 months back however not any more with Loki (2.4.2) and grafana being (8.3.6 or 8.4). This is with trying to add loki as prometheus datasource. Any help would be appreciated.

'Error reading Prometheus: server_error: server error: 500'

@zquaisar having the same issue, with loki 2.4.2 version on distributed mode, and grafana 8 .

Please note that Loki has its own data source in Grafana since a few versions ago. Try using it instead of Prometheus.

@DylanGuedes its doesn't work with the version of Grafana version 8.x. doesn't recognise the Loki endpoint '/loki'

same issue for me with grafana version 7/8 and loki 2.6.1