At SoundCloud we work around the problem by using an extra exporter which exports all pod labels in a separate metric and we then join these together in our queries. This is similar to the approach described in @brian-brazil's blog post about machine role labels. Exporting a lot labels per metric makes it more difficult to work with them. So this might even be the more desirable approach.

@fabxc @kubernetes/sig-instrumentation We should discuss this need in our next meeting.

I propose a implementing a workaround, so that cAdvisor can directly get the labels from the kublet and export them accordingly.

IIUC this is the approach @vishh has advocated as well, particularly as part of the work for #18770.

http://www.robustperception.io/exposing-the-software-version-to-prometheus/ is a slightly more relevant version of that blog post.

@grobie @brian-brazil Although we're using Prometheus' exposition format, we do have to make sure that any decisions we make around labelling don't restrict/negatively affect other consumers. If this was a Prometheus only decision then I would 100% agree with you though.

Although we're using Prometheus' exposition format, we do have to make sure that any decisions we make around labelling don't restrict/negatively affect other consumers. If this was a Prometheus only decision then I would 100% agree with you though.

I think its a reasonable step of normalization to constrain labels to the minimum identity-giving set in the exposition. The consumer is free to denormalize again for its own purposes – in Prometheus this happens to be at query time. But other systems can easily do so on write.
Going the other direction is generally harder for the consumer while bloating the exposed metrics.

Of course it's saner if it's happening in the same metric set. But #18770 aims at just that.

@grobie Any chance of open sourcing that?

My idea was that kubelet should expose an extension API that let's
monitoring agents figure out the list of pods on the node along with
detailed runtime information like container & image identifiers, etc.
cAdvisor, being a monitoring agent, can then use this API to come up with
pod level metrics and metadata for k8s pods.

On Wed, Sep 14, 2016 at 1:27 PM, Steffen Hanikel notifications@github.com
wrote:

@grobie https://github.com/grobie Any chance of open sourcing that?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32326 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKPZqoTMWEa_8odh1h_EMsVjWw8FZks5qqFiwgaJpZM4J4h6h
.

@hanikesn @grobie FYI just added support for exporting a pod-label-metric for kube-api-exporter: tomwilkie/kube-api-exporter#9

(edited) I also wrote a blog post on how to do the join in Prometheus - https://www.weave.works/aggregating-pod-resource-cpu-memory-usage-arbitrary-labels-prometheus/

is any chance that this will be implemented in the next release ?

Thanks a lot!

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

/remove-lifecycle rotten

cc @dchen1107

This would be very useful.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@hanikesn any news on this?

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Note that prometheus/client_golang doesn't require consistent label dimensions anymore, which might come in handy here. See prometheus/client_golang#417 . Also, you can now have unchecked Collectors in a formally correct way.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@brancz Still planned?

@Nowaker the metrics overhaul has been moved to 1.14 but the pull request is already out to make this change!

Yay! Thanks a ton @brancz.

@brancz got a reference for the 1.14 PR?

The PR is here: #69099

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot: Closing this issue.

#69099 was merged, so this is landing in 1.14 🎉

/reopen

@hangyan @brancz I believe closing this issue via #69099 is a misunderstanding. #69099 is about adding the literal pod label to the cadvisor metrics. This issue is about adding the labels that are on the pod as label to the cadvisor metrics.

@alvaroaleman: Reopened this issue.

I see. I think that’s metadata that should be joined onto the metric at query time (this data itself is already available in kube state metrics). More importantly labels on a pod can change over a pod’s lifetime which if we added pod labels to these metrics they would be marked stale and produce a new time-series which wouldn’t reflect the reality.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot: Closing this issue.

I partially solved my case: Needs to have 'deployment' or 'app' label as a filter for pods names
grafana/grafana#28447

But how nice would it be, if cAdvisor could forward an 'app' label along with metrics...

/reopen

please export all Pod labels ('app', 'deployment', etc...)

like this
__meta_kubernetes_pod_label_app="schema-registry-ui"

@sirkubax: You can't reopen an issue/PR unless you authored it or you are a collaborator.

Any updates on this?

/reopen
Is there any plan to support add pod label within cadvisor?

@woodliu: You can't reopen an issue/PR unless you authored it or you are a collaborator.

Hot to ?