Skip to content

How to monitor Nginx ingress controller? #2119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ArjonBu opened this issue Nov 12, 2018 · 12 comments
Closed

How to monitor Nginx ingress controller? #2119

ArjonBu opened this issue Nov 12, 2018 · 12 comments

Comments

@ArjonBu
Copy link

ArjonBu commented Nov 12, 2018 •
edited
Loading

What did you do?
I am trying to monitor Nginx ingress controller with Prometheus operator.
I have installed Nginx ingress controller using the official helm chart and I have enabled metrics, stats, and I have enabled the following annotations for metrics:

prometheus.io/scrape: "true"
prometheus.io/port: "10254"

The ingress controller will create the following services:

nginx-ingress-controller                       LoadBalancer   100.67.71.32     myloadbalancer.elb.amazonaws.com   80:30794/TCP,443:30883/TCP   2d
nginx-ingress-controller-metrics               ClusterIP      100.66.15.57     <none>                                                                    9913/TCP                     2d
nginx-ingress-controller-stats                 ClusterIP      100.68.130.105   <none>                                                                    18080/TCP                    2d
nginx-ingress-default-backend                  ClusterIP      100.65.98.230    <none>                                                                    80/TCP                       2d

Environment

  • Prometheus Operator version:

v0.25.0

  • Kubernetes version information:

v1.10.6

  • Kubernetes cluster kind:

Kops, AWS EKS

  • Manifests:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress-controller-metrics
  labels:
    app: nginx-ingress
spec:
  endpoints:
  - interval: 30s
    port: metrics
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  namespaceSelector:
    matchNames:
    - default
  • Prometheus Operator Logs:
level=info ts=2018-11-12T09:33:19.529074046Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:27.037554394Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:34.543325319Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:42.052934524Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:49.563343133Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:57.065484489Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:34:04.572233431Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus

I think I am doing something wrong with the ServiceMonitor manifest file. Can someone help?

Thanks in advance
@metalmatze
Copy link
Member

Is everything running in the default namespace?

Looking at the services you posted, could make sure that the metrics port is 9913 and the labels are the same?

nginx-ingress-controller-metrics               ClusterIP      100.66.15.57     <none>                                                                    9913/TCP                     2d

Feel free to paste your output of kubectl -n default describe service nginx-ingress-controller-metrics here so we can debug further.

@ArjonBu
Copy link
Author

ArjonBu commented Nov 12, 2018

@metalmatze Yes, everything is running in the default namespace.

Here is the output of the service:

Name:              nginx-ingress-controller-metrics
Namespace:         default
Labels:            app=nginx-ingress
                   chart=nginx-ingress-0.30.0
                   component=controller
                   heritage=Tiller
                   release=nginx-ingress
Annotations:       prometheus.io/port: 10254
                   prometheus.io/scrape: true
Selector:          app=nginx-ingress,component=controller,release=nginx-ingress
Type:              ClusterIP
IP:                100.66.15.57
Port:              metrics  9913/TCP
TargetPort:        metrics/TCP
Endpoints:         100.96.12.2:10254
Session Affinity:  None
Events:            <none>

@ArjonBu
Copy link
Author

ArjonBu commented Nov 15, 2018

Any update on this?

@metalmatze
Copy link
Member

I still do not really get why you have Port: metrics 9913/TCP when it everywhere says 10254. Are you able to change the port in the Service to 10254 as well? Else I can't really see anything wrong.

@ArjonBu
Copy link
Author

ArjonBu commented Nov 16, 2018

I changed the port to 10254/tcp and it still doesn't show anything. Is prometheus config automatically reloaded after every servicemonitor change?

@squat
Copy link
Contributor

squat commented Nov 16, 2018

@ArjonBu yes, the Prometheus configuration is automatically re-generated and reloaded after every change to a ServiceMonitor that it is watching. It is possible that your Prometheus is not selecting this ServiceMonitor for some reason. Can you please share the Prometheus manifest?

@ArjonBu
Copy link
Author

ArjonBu commented Nov 16, 2018

@squat I installed prometheus operator using the official helm chart. If you meant the serviceoperator. I have pasted it in the first post.

@squat
Copy link
Contributor

squat commented Nov 16, 2018

@ArjonBu the helm charts are not maintained by this team, plus there are many ways to configure the chart to have different behaviors. Can you please share the output of kubectl get prometheus -n default prometheus-operator-prometheus -o yaml

@ArjonBu
Copy link
Author

ArjonBu commented Nov 16, 2018

@squat Here is it:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  clusterName: ""
  creationTimestamp: 2018-11-08T10:12:35Z
  generation: 1
  labels:
    app: prometheus-operator-prometheus
    chart: prometheus-operator-0.1.11
    heritage: Tiller
    release: prometheus-operator
  name: prometheus-operator-prometheus
  namespace: default
  resourceVersion: "189947"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/default/prometheuses/prometheus-operator-prometheus
  uid: d061bf3c-e33e-11e8-b3d6-022ffae4fd40
spec:
  alerting:
    alertmanagers:
    - name: prometheus-operator-alertmanager
      namespace: default
      pathPrefix: /
      port: web
  baseImage: quay.io/prometheus/prometheus
  externalUrl: http://prometheus-operator-prometheus.default:9090
  listenLocal: false
  logLevel: info
  paused: false
  replicas: 1
  retention: 10d
  routePrefix: /
  ruleSelector:
    matchLabels:
      app: prometheus-operator
      release: prometheus-operator
  serviceAccountName: prometheus-operator-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: prometheus-operator
  version: v2.4.3

@squat
Copy link
Contributor

squat commented Nov 17, 2018

@ArjonBu thanks, that's super helpful. It appears to me that indeed your Prometheus is not selecting your ServiceMonitors due to mismatched labels. Notice that your Prometheus objects has:

serviceMonitorSelector:
  matchLabels:
    release: prometheus-operator

but the only labels on your Ingress Controller ServiceMonitor are:

app: nginx-ingress

In order to select your Ingress Controller ServiceMonitor, you will need to add:

release: prometheus-operator

to the ServiceMonitor's labels so that it looks like:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress-controller-metrics
  labels:
    app: nginx-ingress
    release: prometheus-operator
spec:
  endpoints:
  - interval: 30s
    port: metrics
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  namespaceSelector:
    matchNames:
    - default

@ArjonBu
Copy link
Author

ArjonBu commented Nov 19, 2018

Works! Thank you

@squat
Copy link
Contributor

squat commented Nov 19, 2018

@ArjonBu, I’m very glad this solved the issue for you. I’ll close this now since everything is working.

@squat squat closed this as completed Nov 19, 2018
@ravvereddy ravvereddy mentioned this issue Feb 10, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@metalmatze @ArjonBu @squat