Closed
Description
What would you like to be added:
I am trying to use build-in cadvisor to monitor the containers. I scrape the metrics from api-server:http://localhost:8080/api/v1/nodes/minikube/proxy/metrics/cadvisor
and get some metrics like this:
container_fs_reads_merged_total{container="nginx",container_name="nginx",device="/dev/sda1",id="/kubepods/besteffort/pod44db6138-9989-11e9-acf7-080027bfe48d/628e7f1886add998474f618d50c8b6564ead95cde28da4f07796c1f8fc59830b",image="nginx@sha256:96fb261b66270b900ea5a2c17a26abbfabe95506e73c3a3c65869a6dbe83223a",name="k8s_nginx_nginx-deployment-64fc4c755d-w9gws_default_44db6138-9989-11e9-acf7-080027bfe48d_1",namespace="default",pod="nginx-deployment-64fc4c755d-w9gws",pod_name="nginx-deployment-64fc4c755d-w9gws"} 0 1562129787870
The docker inspect
command prints the container labels:
"Labels": {
"annotation.io.kubernetes.container.hash": "b6b875a4",
"annotation.io.kubernetes.container.ports": "[{\"containerPort\":80,\"protocol\":\"TCP\"}]",
"annotation.io.kubernetes.container.restartCount": "1",
"annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"annotation.io.kubernetes.container.terminationMessagePolicy": "File",
"annotation.io.kubernetes.pod.terminationGracePeriod": "30",
"io.kubernetes.container.logpath": "/var/log/pods/default_nginx-deployment-64fc4c755d-w9gws_44db6138-9989-11e9-acf7-080027bfe48d/nginx/1.log",
"io.kubernetes.container.name": "nginx",
"io.kubernetes.docker.type": "container",
"io.kubernetes.pod.name": "nginx-deployment-64fc4c755d-w9gws",
"io.kubernetes.pod.namespace": "default",
"io.kubernetes.pod.uid": "44db6138-9989-11e9-acf7-080027bfe48d",
"io.kubernetes.sandbox.id": "1786d56f1f53bbd4852e2ad15550739c77a9648d696ba5997f3d8298dae910e8",
"maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
},
It is reasonable to remove the internal labels(annotation.io.xxxx, io.kubernetes.xxx), but how can i expose my custom container labels such as “maintainer” to cadvisor?
Why is this needed:
I need container labels to do aggregation in promethues.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Activity
DrmagicE commentedon Jul 5, 2019
/sig node
irajdeep commentedon Aug 20, 2019
Sounds somewhat related?
#79702
dchen1107 commentedon Aug 20, 2019
cc/ @dashpole who has been looking into this lately ...
dashpole commentedon Aug 21, 2019
As a workaround, you can run cAdvisor as a daemonset. It is generally recommended not to add container labels as prometheus labels as the huge number of labels added is expensive for prometheus to store.
DrmagicE commentedon Aug 22, 2019
@dashpole Thanks. Deamonset can solve my problems, but I think it would be more convenient if kubernetes can support this natively. It is a little weird to have two cadvisor doing most of the same thing.