Prometheus exporter for custom eBPF metrics.
This is a fork of cloudflare/ebpf-exporter, improve for kubernetes node environment.
Motivation of this exporter is to allow you to write eBPF code and export metrics that are not otherwise accessible from the Linux kernel.
eBPF was described by Ingo Molnár as:
One of the more interesting features in this cycle is the ability to attach eBPF programs (user-defined, sandboxed bytecode executed by the kernel) to kprobes. This allows user-defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively.
An easy way of thinking about this exporter is bcc tools as prometheus metrics:
- https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md
- http://www.brendangregg.com/ebpf.html
To deploy, you need to have kernel-devel installed:
yum install kernel-devel -y
To build, you need to have libbcc installed:
You can use pre-compiled binary from Releases:
That still requires you to have libbcc. To build release binaries yourself:
$ make release-binaries
To build a package from latest sources:
$ go get -u -v github.com/ahas-sigs/kube-ebpf-exporter/...
To run with kernel-3.10 config (you need root privileges, unser kernel 3.10 version, such as centos 7.6):
$ ~/go/bin/kube-ebpf_exporter --config.file=src/github.com/ahas-sigs/kube-ebpf-exporter/examples/ahas-kernel-3.10.yaml
See benchmark directory to get an idea of how low ebpf overhead is.
Currently the only supported way of getting data out of the kernel is via maps (we call them tables in configuration). See:
See examples section for real world examples.
If you have examples you want to share, please feel free to open a PR.
Skip to format to see the full specification.
You can find additional examples in examples directory.
Unless otherwise specified, all examples are expected to work on centos kernel 3.10.0-1062, which is the current kernerl version of Centos 7.
# HELP ebpf_exporter_tcp_connect_latency_seconds Tcp connect latency histogram
# TYPE ebpf_exporter_tcp_connect_latency_seconds histogram
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="1e-06"} 0
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="2e-06"} 0
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="4e-06"} 0
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="8e-06"} 0
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="1.6e-05"} 0
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="3.2e-05"} 0
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="6.4e-05"} 2
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.000128"} 6
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.000256"} 8
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.000512"} 9
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.001024"} 9
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.002048"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.004096"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.008192"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.016384"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.032768"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.065536"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.131072"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.262144"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="0.524288"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="1.048576"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="2.097152"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="4.194304"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="8.388608"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="16.777216"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="33.554432"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="67.108864"} 10
ebpf_exporter_tcp_connect_latency_seconds_bucket{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127",le="+Inf"} 10
ebpf_exporter_tcp_connect_latency_seconds_sum{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127"} 0.002206
ebpf_exporter_tcp_connect_latency_seconds_count{app_container="coredns",app_namespace="kube-system",node_id="localhost",subnet="127"} 10