Please note the hack/ directory that you're executing this from. The kube-prometheus stack expects you to have a properly secured setup, which allows authenticating with ServiceAccount tokens and authorizes against RBAC roles. What this concretely means for a minikube cluster for example is documented here: https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/hack/cluster-monitoring/minikube-deploy#L3-L12

Basically your cluster needs to be RBAC enabled and these two kubelet flags need to be enabled:

• --authentication-token-webhook=true
• --extra-config=kubelet.authorization-mode=Webhook

Feel free to ask any further questions, but this is not an issue with the Prometheus operator or kube-prometheus stack, so I'm closing this here.

FWIW this seems to resolve the issue - kubernetes/kubernetes#44330 (comment)

for future googlers, changing the kubelet ServiceMonitor to look for the http endpoints on port 10255 worked for me:

(prometheus-k8s-service-monitor-kubelet.yaml for the hack/ example)

port: https-metrics changes to port: http-metrics (in code here)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet
  labels:
    k8s-app: kubelet
spec:
  jobLabel: k8s-app
  endpoints:
  - port: http-metrics
    scheme: http
    interval: 30s
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  - port: http-metrics
    scheme: http
    path: /metrics/cadvisor
    interval: 30s
    honorLabels: true
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  selector:
    matchLabels:
      k8s-app: kubelet
  namespaceSelector:
    matchNames:
    - kube-system

Running on GKE, I solved a similar issue with adding --set exporter-kubelets.https=false to my helm install command.
See comment in helm/exporter-kubelets/values.yaml:

# Set to false for GKE
https: true

--set exporter-kubelets.https=false didnt solve the problem

I had the same problem, using GKE. Solved it by updating the kube-prometheus-exporter-kubelets ServiceMonitor resource definition, from HTTPS to HTTP, as @vsinha suggested, with this one-liner (update the namespace accordingly):

$ kubectl -n monitoring get servicemonitor kube-prometheus-exporter-kubelets -o yaml | sed 's/https/http/' | kubectl replace -f -
servicemonitor.monitoring.coreos.com "kube-prometheus-exporter-kubelets" replaced

After this, the Prometheus server was able to scrape the kubelet target as expected.

Also applies to AKS, I switched the ServiceMonitor to http as workaround.

Apparently this is known behaviour. See the kube-prometheus here: https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/docs/GKE-cadvisor-support.md

Confirming you still need to use this in the kubelets ServiceMonitor for GKE: 1.11.5-gke.5

scheme: http
port: http-metrics

Not sure if this is the right place to leave this, but adding it here as I hit a similar issue and this was the first result. I was using AWS/EKS, but I think this has more to do with k8s v1.11. It seems now the read-only port is disabled by default now. I had to re-enable this on all my nodes.

I am using a launch configuration and ASGs, so that would look something like this:

#!/bin/bash -xe

# Bootstrap and join the cluster
/etc/eks/bootstrap.sh --b64-cluster-ca  'CERT_HERE' --apiserver-endpoint 'ENDPOINT_HERE' --kubelet-extra-args '--read-only-port=10255' 'CLUSTERNAME_HERE'

after this change all worked as expected.

Another thing that made the issue more obvious that in prometheus under /targets you could see connection being refused along with data missing in grafana.

It took way to long for me to find this, so hopefully it helps someone else out.

@gb-ckedzierski, since you're modifying the kubelet config anyway, you should leave the --read-only-port disabled and use the secure port with these flags:

--authentication-token-webhook=true
--extra-config=kubelet.authorization-mode=Webhook

We tried the way @vsinha suggested, and we encountered another problem. The cadvisor metrics always return "INVALID" is not a valid start token error. Below are the warning log in prometheous v2.6.1.

level=warn ts=2019-01-16T09:24:01.100534348Z caller=scrape.go:835 component="scrape manager" scrape_pool=monitoring/kube-prometheus-exporter-kubelets/1 target=http://10.140.0.28:4194/metrics/cadvisor msg="append failed" err="\"INVALID\" is not a valid start token"

@markya0616 you're requesting a different port than @vsinha. What you're requesting is the standalone cadvisor port (which has been removed in 1.13 I believe).

In the latest helm chart, set kubelet.serviceMonitor.https=false:

kubelet:
  enabled: true
  serviceMonitor:
    https: false

This forces kubelet exporter to scrape the http-metric endpoint, which should solve the problem.

EDIT: s/http/https/

@kevinjqiu thanks for the pointer! btw, your snippet there is incorrect. s/http/https/

@jwineinger Thanks. Updated.

I just confirmed that on AKS, --authentication-token-webhook is set to false the default from kubelet.

Azure/AKS#1087

I still had to set @kevinjqiu values to make this work, though the latest v8.0.0 stable helm chart declares https: false as default value.

In case anyone comes across this issue again with AKS and k8s 1.18.4.
With the chart stable/prometheus-operator in version 9.2.0 removing the suggested change

- kubelet:
-   serviceMonitor:
-     https: false

fixes the issue.