Skip to content

cadvisor in kubelet should expose container labels #79702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
DrmagicE opened this issue Jul 3, 2019 · 5 comments
Closed

cadvisor in kubelet should expose container labels #79702

DrmagicE opened this issue Jul 3, 2019 · 5 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@DrmagicE
Copy link

DrmagicE commented Jul 3, 2019

What would you like to be added:
I am trying to use build-in cadvisor to monitor the containers. I scrape the metrics from api-server：http://localhost:8080/api/v1/nodes/minikube/proxy/metrics/cadvisor and get some metrics like this:

container_fs_reads_merged_total{container="nginx",container_name="nginx",device="/dev/sda1",id="/kubepods/besteffort/pod44db6138-9989-11e9-acf7-080027bfe48d/628e7f1886add998474f618d50c8b6564ead95cde28da4f07796c1f8fc59830b",image="nginx@sha256:96fb261b66270b900ea5a2c17a26abbfabe95506e73c3a3c65869a6dbe83223a",name="k8s_nginx_nginx-deployment-64fc4c755d-w9gws_default_44db6138-9989-11e9-acf7-080027bfe48d_1",namespace="default",pod="nginx-deployment-64fc4c755d-w9gws",pod_name="nginx-deployment-64fc4c755d-w9gws"} 0 1562129787870

The docker inspect command prints the container labels:

"Labels": {
                "annotation.io.kubernetes.container.hash": "b6b875a4",
                "annotation.io.kubernetes.container.ports": "[{\"containerPort\":80,\"protocol\":\"TCP\"}]",
                "annotation.io.kubernetes.container.restartCount": "1",
                "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.container.logpath": "/var/log/pods/default_nginx-deployment-64fc4c755d-w9gws_44db6138-9989-11e9-acf7-080027bfe48d/nginx/1.log",
                "io.kubernetes.container.name": "nginx",
                "io.kubernetes.docker.type": "container",
                "io.kubernetes.pod.name": "nginx-deployment-64fc4c755d-w9gws",
                "io.kubernetes.pod.namespace": "default",
                "io.kubernetes.pod.uid": "44db6138-9989-11e9-acf7-080027bfe48d",
                "io.kubernetes.sandbox.id": "1786d56f1f53bbd4852e2ad15550739c77a9648d696ba5997f3d8298dae910e8",
                "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
            },

It is reasonable to remove the internal labels(annotation.io.xxxx, io.kubernetes.xxx), but how can i expose my custom container labels such as “maintainer” to cadvisor？
Why is this needed:
I need container labels to do aggregation in promethues.
@DrmagicE DrmagicE added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 3, 2019
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 3, 2019
@DrmagicE
Copy link
Author

DrmagicE commented Jul 5, 2019

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 5, 2019
@irajdeep
Copy link
Contributor

Sounds somewhat related?
#79702

@dchen1107
Copy link
Member

cc/ @dashpole who has been looking into this lately ...

@dashpole
Copy link
Contributor

As a workaround, you can run cAdvisor as a daemonset. It is generally recommended not to add container labels as prometheus labels as the huge number of labels added is expensive for prometheus to store.

@DrmagicE
Copy link
Author

@dashpole Thanks. Deamonset can solve my problems, but I think it would be more convenient if kubernetes can support this natively. It is a little weird to have two cadvisor doing most of the same thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dashpole @irajdeep @dchen1107 @DrmagicE @k8s-ci-robot