Do you mind rewriting the issue to abide by our standard issue template?

I used the issue template, I just deleted some of the parts that I don't think that apply to this issue. But I will add them. Thanks for the help.

Have the same error with Grafana 6.7.1 and Loki 1.3
Call stack is:

react-dom.production.min.js:196 TypeError: Cannot read property 'slice' of null
at t.n.rowsToRender (LiveLogs.tsx:132)
at t.value (LiveLogs.tsx:151)
at Ka (react-dom.production.min.js:181)
at Xa (react-dom.production.min.js:180)
at Ou (react-dom.production.min.js:261)
at vs (react-dom.production.min.js:230)
at ps (react-dom.production.min.js:229)
at os (react-dom.production.min.js:223)
at react-dom.production.min.js:121
at e.unstable_runWithPriority (scheduler.production.min.js:18)

Same issue here, grafana 6.7.1 and loki 1.3, logcli is running fine in tail mode

Same issue:
Grafana 6.6.2
Loki 1.3

Hi, when I try this with Grafana 6.7.2 the live logs for Loki works for me. Has this been resolved maybe?

I just tested 6.7.2 but the issue is still happening. From what I know it does not happen to everyone.

Maybe you need to update Loki? What version of Loki are you using? Also try latest version of Grafana

Maybe you have proxy in front of Grafana that does not support websockets?

Tried latest version of Grafana and Loki with no success.

I do have a proxy in front of Grafana and also a proxy between Grafana and Loki, but this setup is working great with Grafana 6.5.3, so proxies are not the problem. It breaks from Grafana 6.6.0. Maybe some change was introduced in that version that is not compatible with our setup.

Do you recall any changes introduced in this version that could be the source of the problem? Maybe endpoint changes or any other change? Not saying a bug, but maybe just a breaking change that we should be aware of.

I found the issue!! It was an endpoint change like I said in the previous message.

Now Grafana is requesting this Loki's endpoint:
/loki/api/v1/tail
instead of the older:
/api/prom/tail

So any setup in the proxies related to the older version, like a proxy_set_header Upgrade, would not be applied from Grafana 6.6.0

Hello @danieldabate! I am glad that you were able to solve the problem. 🚀 I am going to close this issue as in the next release we are are going to stop supporting legacy Loki endpoints (before v1). As it is a breaking change, it is going to be communicated in the release notes.

Thanks @danieldabate, you saved my some time.

@ivanahuckova 7.0.0-beta3 still this issue

nginx-proxy
loki 1.4.1

Live mode requires websocket.
If nginx is used, websocket support is required
https://www.nginx.com/blog/websocket-nginx/

@s5unty
Thank you, your link helped

Just to help those coming here from a fresh Grafana and Loki deployment via Helm, when you add Loki to your Data Sources, check your logs on the loki pod and you will likely see errors about the Connection or Upgrade headers. You need to add two custom headers in the config for the Loki Data Source in Grafana.

• Connection: Upgrade
• Upgrade: websocket

I am having this issue no proxy in the middle (between grafan and loki) added the two custom headers the issue persists.

I'm using ambassador Mappings, doing this worked for me, which means. for those not working, try allowing WebSocket connection upgrade. to their controllers/servers.

Custom headers didn't solve it for me but here is the config:

datasources:
  - name: Loki
    type: loki
    url: "http://loki.local:3100"
    access: proxy
    version: 1
    org_id: 1
    isDefault: false
    jsonData:
      timeInterval: 15s
      httpHeaderName1: Connection
      httpHeaderName2: Upgrade
    secureJsonData:
      httpHeaderValue1: Upgrade
      httpHeaderValue2: websocket

I'm getting parse error : syntax error: unexpected $end from GET /api/datasources/proxy/95/loki/api/v1/query_range and WebSocketSubject.js:96 WebSocket connection to 'wss://loki.local/api/datasources/proxy/95/loki/api/v1/tail?query=%7Bjob%3D%22systemd-journal%22%7D' failed:

we are having exact same issues when X-Scope-OrgID header contains more than 1 tenant id.
Grafana v9.5.3
Loki v2.8.2

Custom headers didn't solve it for me but here is the config:

datasources:
  - name: Loki
    type: loki
    url: "http://loki.local:3100"
    access: proxy
    version: 1
    org_id: 1
    isDefault: false
    jsonData:
      timeInterval: 15s
      httpHeaderName1: Connection
      httpHeaderName2: Upgrade
    secureJsonData:
      httpHeaderValue1: Upgrade
      httpHeaderValue2: websocket

I'm getting parse error : syntax error: unexpected $end from GET /api/datasources/proxy/95/loki/api/v1/query_range and WebSocketSubject.js:96 WebSocket connection to 'wss://loki.local/api/datasources/proxy/95/loki/api/v1/tail?query=%7Bjob%3D%22systemd-journal%22%7D' failed:

This is a completely different error than what the custom headers are supposed to solve.

Custom headers only solve the issue when the error states that it is a Connection or Upgrade header issue.

I was able to fix this by enabling WebSocket on the Grafana side along with the following

httpHeaderName1: Connection
httpHeaderName2: Upgrade
httpHeaderValue1: Upgrade
httpHeaderValue2: websocket

For what its worth, my grafana instance is behind HTTPProxy with Contour.

Here's for nginx (https://www.civo.com/learn/using-websockets-with-ingress-controller)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: web-socket
 annotations:
  kubernetes.io/ingress.class: nginx
  nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  nginx.ingress.kubernetes.io/server-snippets: |
   location / {
    proxy_set_header Upgrade $http_upgrade;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header Connection "upgrade";
    proxy_cache_bypass $http_upgrade;
    }
...

My HTTPProxy is as follows.

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: grafana
  namespace: grafana
spec:
  virtualhost:
    fqdn: private.com
    tls:
      secretName: cert-tls
  routes:
  - services:
    - name: grafana-operator-grafana-service
      port: 3000
    enableWebsockets: true