Skip to content

How to monitor Nginx ingress controller? #2119

New issue
New issue
Closed
Closed
How to monitor Nginx ingress controller?#2119
@ArjonBu

Description

@ArjonBu
ArjonBu
opened on Nov 12, 2018 · edited by ArjonBu

What did you do?
I am trying to monitor Nginx ingress controller with Prometheus operator.
I have installed Nginx ingress controller using the official helm chart and I have enabled metrics, stats, and I have enabled the following annotations for metrics:

prometheus.io/scrape: "true"
prometheus.io/port: "10254"

The ingress controller will create the following services:

nginx-ingress-controller                       LoadBalancer   100.67.71.32     myloadbalancer.elb.amazonaws.com   80:30794/TCP,443:30883/TCP   2d
nginx-ingress-controller-metrics               ClusterIP      100.66.15.57     <none>                                                                    9913/TCP                     2d
nginx-ingress-controller-stats                 ClusterIP      100.68.130.105   <none>                                                                    18080/TCP                    2d
nginx-ingress-default-backend                  ClusterIP      100.65.98.230    <none>                                                                    80/TCP                       2d

Environment

  • Prometheus Operator version:

v0.25.0

  • Kubernetes version information:

v1.10.6

  • Kubernetes cluster kind:

Kops, AWS EKS

  • Manifests:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress-controller-metrics
  labels:
    app: nginx-ingress
spec:
  endpoints:
  - interval: 30s
    port: metrics
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  namespaceSelector:
    matchNames:
    - default
  • Prometheus Operator Logs:
level=info ts=2018-11-12T09:33:19.529074046Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:27.037554394Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:34.543325319Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:42.052934524Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:49.563343133Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:33:57.065484489Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus
level=info ts=2018-11-12T09:34:04.572233431Z caller=operator.go:956 component=prometheusoperator msg="sync prometheus" key=default/prometheus-operator-prometheus

I think I am doing something wrong with the ServiceMonitor manifest file. Can someone help?

Thanks in advance

Activity

metalmatze

metalmatze commented on Nov 12, 2018

@metalmatze
metalmatze
on Nov 12, 2018
Member

Is everything running in the default namespace?

Looking at the services you posted, could make sure that the metrics port is 9913 and the labels are the same?

nginx-ingress-controller-metrics               ClusterIP      100.66.15.57     <none>                                                                    9913/TCP                     2d

Feel free to paste your output of kubectl -n default describe service nginx-ingress-controller-metrics here so we can debug further.

ArjonBu

ArjonBu commented on Nov 12, 2018

@ArjonBu
ArjonBu
on Nov 12, 2018
Author

@metalmatze Yes, everything is running in the default namespace.

Here is the output of the service:

Name:              nginx-ingress-controller-metrics
Namespace:         default
Labels:            app=nginx-ingress
                   chart=nginx-ingress-0.30.0
                   component=controller
                   heritage=Tiller
                   release=nginx-ingress
Annotations:       prometheus.io/port: 10254
                   prometheus.io/scrape: true
Selector:          app=nginx-ingress,component=controller,release=nginx-ingress
Type:              ClusterIP
IP:                100.66.15.57
Port:              metrics  9913/TCP
TargetPort:        metrics/TCP
Endpoints:         100.96.12.2:10254
Session Affinity:  None
Events:            <none>
ArjonBu

ArjonBu commented on Nov 15, 2018

@ArjonBu
ArjonBu
on Nov 15, 2018
Author

Any update on this?

metalmatze

metalmatze commented on Nov 15, 2018

@metalmatze
metalmatze
on Nov 15, 2018
Member

I still do not really get why you have Port: metrics 9913/TCP when it everywhere says 10254. Are you able to change the port in the Service to 10254 as well? Else I can't really see anything wrong.

ArjonBu

ArjonBu commented on Nov 16, 2018

@ArjonBu
ArjonBu
on Nov 16, 2018
Author

I changed the port to 10254/tcp and it still doesn't show anything. Is prometheus config automatically reloaded after every servicemonitor change?

squat

squat commented on Nov 16, 2018

@squat
squat
on Nov 16, 2018
Contributor

@ArjonBu yes, the Prometheus configuration is automatically re-generated and reloaded after every change to a ServiceMonitor that it is watching. It is possible that your Prometheus is not selecting this ServiceMonitor for some reason. Can you please share the Prometheus manifest?

ArjonBu

ArjonBu commented on Nov 16, 2018

@ArjonBu
ArjonBu
on Nov 16, 2018
Author

@squat I installed prometheus operator using the official helm chart. If you meant the serviceoperator. I have pasted it in the first post.

squat

squat commented on Nov 16, 2018

@squat
squat
on Nov 16, 2018
Contributor

@ArjonBu the helm charts are not maintained by this team, plus there are many ways to configure the chart to have different behaviors. Can you please share the output of kubectl get prometheus -n default prometheus-operator-prometheus -o yaml

ArjonBu

ArjonBu commented on Nov 16, 2018

@ArjonBu
ArjonBu
on Nov 16, 2018
Author

@squat Here is it:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  clusterName: ""
  creationTimestamp: 2018-11-08T10:12:35Z
  generation: 1
  labels:
    app: prometheus-operator-prometheus
    chart: prometheus-operator-0.1.11
    heritage: Tiller
    release: prometheus-operator
  name: prometheus-operator-prometheus
  namespace: default
  resourceVersion: "189947"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/default/prometheuses/prometheus-operator-prometheus
  uid: d061bf3c-e33e-11e8-b3d6-022ffae4fd40
spec:
  alerting:
    alertmanagers:
    - name: prometheus-operator-alertmanager
      namespace: default
      pathPrefix: /
      port: web
  baseImage: quay.io/prometheus/prometheus
  externalUrl: http://prometheus-operator-prometheus.default:9090
  listenLocal: false
  logLevel: info
  paused: false
  replicas: 1
  retention: 10d
  routePrefix: /
  ruleSelector:
    matchLabels:
      app: prometheus-operator
      release: prometheus-operator
  serviceAccountName: prometheus-operator-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: prometheus-operator
  version: v2.4.3
squat

squat commented on Nov 17, 2018

@squat
squat
on Nov 17, 2018
Contributor

@ArjonBu thanks, that's super helpful. It appears to me that indeed your Prometheus is not selecting your ServiceMonitors due to mismatched labels. Notice that your Prometheus objects has:

serviceMonitorSelector:
  matchLabels:
    release: prometheus-operator

but the only labels on your Ingress Controller ServiceMonitor are:

app: nginx-ingress

In order to select your Ingress Controller ServiceMonitor, you will need to add:

release: prometheus-operator

to the ServiceMonitor's labels so that it looks like:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-ingress-controller-metrics
  labels:
    app: nginx-ingress
    release: prometheus-operator
spec:
  endpoints:
  - interval: 30s
    port: metrics
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  namespaceSelector:
    matchNames:
    - default
ArjonBu

ArjonBu commented on Nov 19, 2018

@ArjonBu
ArjonBu
on Nov 19, 2018
Author

Works! Thank you

squat

squat commented on Nov 19, 2018

@squat
squat
on Nov 19, 2018
Contributor

@ArjonBu, I’m very glad this solved the issue for you. I’ll close this now since everything is working.

squat
closed this as completedon Nov 19, 2018
ravvereddy
mentioned this on Feb 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @metalmatze@ArjonBu@squat

        Issue actions
          How to monitor Nginx ingress controller? · Issue #2119 · prometheus-operator/prometheus-operator