Description
As of Kubernetes v1.1, and possibly v1.2 as well, cAdvisor is linked into kubelet.
Kubelet groups detailed cAdvisor metrics into pods and nodes and exposes them via REST endpoints (/stats)
Monitoring is resource intensive. There are currently several issues filed against kubelet around resource usage which are mostly related to cAdvisor's resource consumption.
Support for custom metrics adds significant resource overhead to kubelet.
Not all users of kubernetes need cAdvisor for monitoring.
Kubelet will need just a few metrics for its functionality. Specifically, cpu usage, memory usage, and filesystem usage for now. These metrics are needed for scheduling and other cluster functionalities.
By separating out cAdvisor monitoring agent from kubelet, users can choose to run cAdvisor as a standalone container and get a lot more detailed metrics.
This split will let us optimize resource impact of kubelet and cadvisor independently.
cAdvisor can expose potentially expensive metrics as well.
For the basic metrics that kubelet needs, it will embed specific libraries from cAdvisor and compute those metrics internally.
cAdvisor needs to be enhanced to support pods. To make this easier, kubelet should expose the pods that it is currently managing via some API. (#9473 talks about one such API)
cc @dchen1107 @timstclair @kubernetes/goog-node @jimmidyson @jszczepkowski
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
ncdc commentedon Dec 16, 2015
cc @kubernetes/rh-cluster-infra
dchen1107 commentedon Dec 17, 2015
The issue is filed based on several offline design discussions including several attempts of improving Kubelet Metrics API made by @timstclair (#15691, #15862, #16621, #17708, #18544, ...).
cc/ @erictune
huang195 commentedon Dec 18, 2015
if we're going forward with collecting the non-core metrics using a separate cadvisor pod, how will the metrics information be exposed to collectors such as heapster? Will kubelet provide a directory service that helps heapster find where cadvisor pod can be accessed or will kubelet act as a middle-man (similar to the way the code works today) that gets metrics from cadvisor and servicing them out to heapster?
vishh commentedon Dec 18, 2015
Heapster will have to get metrics from cAdvisor directly. Heapster can
query the IP:Port information for all the cAdvisor's in the
cluster(essentially one per node), and can collect metrics using cAdvisor's
IP directly.
On Fri, Dec 18, 2015 at 2:58 PM, Hai Huang notifications@github.com wrote:
jimmidyson commentedon Dec 18, 2015
So are you expecting cadvisor pod to be a daemon set?
vishh commentedon Dec 18, 2015
Yes
On Fri, Dec 18, 2015 at 3:07 PM, Jimmi Dyson notifications@github.com
wrote:
huang195 commentedon Dec 19, 2015
How will heapster find the port that cadvisor listens to? If each cadvisor is running in a separate pod, it could be mapped to different host port on different host.
vishh commentedon Dec 19, 2015
Host port is not required. Remember Pods can talk to each other directly ;)
On Fri, Dec 18, 2015 at 4:45 PM, Hai Huang notifications@github.com wrote:
huang195 commentedon Dec 19, 2015
OK, I just saw that you're expecting cadvisor pods to be running as a daemon set, so the host ips are known to the heapster pod collecting the metrics information. Reading how to communicate with a daemon set, I found some information here: https://github.com/kubernetes/kubernetes/blob/master/docs/admin/daemons.md#communicating-with-daemonset-pods, and it says
I take it this means cadvisor will need to listen to the same port on every host, e.g., 4194, so whoever is communicating with it knows how to find it. Or it has to register with DNS.
35 remaining items