Skip to content

cadvisor in kubelet should expose container labels #79702

Closed
@DrmagicE

Description

@DrmagicE

What would you like to be added:
I am trying to use build-in cadvisor to monitor the containers. I scrape the metrics from api-server:http://localhost:8080/api/v1/nodes/minikube/proxy/metrics/cadvisor and get some metrics like this:

container_fs_reads_merged_total{container="nginx",container_name="nginx",device="/dev/sda1",id="/kubepods/besteffort/pod44db6138-9989-11e9-acf7-080027bfe48d/628e7f1886add998474f618d50c8b6564ead95cde28da4f07796c1f8fc59830b",image="nginx@sha256:96fb261b66270b900ea5a2c17a26abbfabe95506e73c3a3c65869a6dbe83223a",name="k8s_nginx_nginx-deployment-64fc4c755d-w9gws_default_44db6138-9989-11e9-acf7-080027bfe48d_1",namespace="default",pod="nginx-deployment-64fc4c755d-w9gws",pod_name="nginx-deployment-64fc4c755d-w9gws"} 0 1562129787870

The docker inspect command prints the container labels:

"Labels": {
                "annotation.io.kubernetes.container.hash": "b6b875a4",
                "annotation.io.kubernetes.container.ports": "[{\"containerPort\":80,\"protocol\":\"TCP\"}]",
                "annotation.io.kubernetes.container.restartCount": "1",
                "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.container.logpath": "/var/log/pods/default_nginx-deployment-64fc4c755d-w9gws_44db6138-9989-11e9-acf7-080027bfe48d/nginx/1.log",
                "io.kubernetes.container.name": "nginx",
                "io.kubernetes.docker.type": "container",
                "io.kubernetes.pod.name": "nginx-deployment-64fc4c755d-w9gws",
                "io.kubernetes.pod.namespace": "default",
                "io.kubernetes.pod.uid": "44db6138-9989-11e9-acf7-080027bfe48d",
                "io.kubernetes.sandbox.id": "1786d56f1f53bbd4852e2ad15550739c77a9648d696ba5997f3d8298dae910e8",
                "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
            },

It is reasonable to remove the internal labels(annotation.io.xxxx, io.kubernetes.xxx), but how can i expose my custom container labels such as “maintainer” to cadvisor?
Why is this needed:
I need container labels to do aggregation in promethues.

Activity

added
kind/featureCategorizes issue or PR as related to a new feature.
on Jul 3, 2019
added
needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
on Jul 3, 2019
DrmagicE

DrmagicE commented on Jul 5, 2019

@DrmagicE
Author

/sig node

added
sig/nodeCategorizes an issue or PR as relevant to SIG Node.
and removed
needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
on Jul 5, 2019
irajdeep

irajdeep commented on Aug 20, 2019

@irajdeep
Contributor

Sounds somewhat related?
#79702

dchen1107

dchen1107 commented on Aug 20, 2019

@dchen1107
Member

cc/ @dashpole who has been looking into this lately ...

dashpole

dashpole commented on Aug 21, 2019

@dashpole
Contributor

As a workaround, you can run cAdvisor as a daemonset. It is generally recommended not to add container labels as prometheus labels as the huge number of labels added is expensive for prometheus to store.

DrmagicE

DrmagicE commented on Aug 22, 2019

@DrmagicE
Author

@dashpole Thanks. Deamonset can solve my problems, but I think it would be more convenient if kubernetes can support this natively. It is a little weird to have two cadvisor doing most of the same thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.sig/nodeCategorizes an issue or PR as relevant to SIG Node.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @dashpole@irajdeep@dchen1107@DrmagicE@k8s-ci-robot

        Issue actions

          cadvisor in kubelet should expose container labels · Issue #79702 · kubernetes/kubernetes