Add Eureka Service Discovery #3369

kangwoo · 2017-10-30T02:28:59Z

Added Eureka Service Discovery.
The Eureka client used a 'https://github.com/kangwoo/go-eureka-client' library

mdlayher · 2017-10-30T16:04:20Z

Relevant issue: #1506.

Doesn't look like it was officially accepted or rejected. Ping @juliusv?

juliusv · 2017-10-30T16:26:26Z

Thanks! Unless something has changed, we still have a moratorium (instated a couple of months ago) for accepting any new SD mechanisms into Prometheus itself at the moment. The reason was that we have too many mechanisms already that are not well-tested, not well-understood, and have too few maintainers for them. We started efforts to introduce integration testing and more documentation (parts of the docs have already happened at least). But I think we still are not accepting new mechanisms into Prometheus proper at this moment. Instead, we would recommend running this outside via the File SD for now and consider it for integration into Prometheus later.

@brian-brazil @cstyan @gouthamve @fabxc Am I correct that this is still the current state?

brian-brazil · 2017-10-30T18:49:42Z

Yes, that's the current state.

From a very brief look, I believe this PR is also hardcoding organisation-specific assumptions.

td83 · 2017-11-09T19:48:41Z

@juliusv - running for example Eureka via SD file is usually not possible in cloud environment where all services are stateless and you cannot mount a file system and share files between Prometheus and bridge application.

godoylucase · 2018-02-22T03:06:29Z

is there any chance to have anyone to take care of this? Would like to have it since we are using netflix oss for our microservices stack

krasi-georgiev · 2018-02-22T09:57:32Z

there is also a discussion for http vs sd to it would be useful get some input regarding this use-case
https://groups.google.com/forum/#!topic/prometheus-developers/JcE5nSbCe4k

ghost · 2018-03-12T14:16:42Z

Until this is merged and released, there is another option to integrate Prometheus with Eureka. I have written an adapter that you can deploy with your Eureka instance. This adapter provides several HTTP endpoints that return information of the registered services and are compatible with the Prometheus' Consul discovery client (consul_sd_config). I have tested it for a couple of month now and it works great. I have put it up on Github: https://github.com/twinformatics/eureka-consul-adapter Feel free to give it a try (and feedback).

krasi-georgiev · 2018-06-05T16:22:20Z

due to lack of man power to review and maintain new SD we are on permanent hold for adding new SD.

As a replacement we have now added an example how to implement a custom SD adapter that converts any SD to json file that can be used by the file SD
https://github.com/prometheus/prometheus/tree/master/documentation/examples/custom-sd

roidelapluie · 2020-07-27T18:51:33Z

@kangwoo are you still interested in this?

I would like to investigate and see how we can integrate this work in Prometheus.

kangwoo · 2020-07-28T12:06:19Z

@kangwoo are you still interested in this?

I would like to investigate and see how we can integrate this work in Prometheus.

yes. How can I help you?

cstyan · 2020-07-28T18:16:29Z

@kangwoo rebasing off master, dealing with any merge conflicts, and ensuring that all tests still pass would be a good place to start :)

roidelapluie · 2020-07-28T22:28:34Z

Maybe also see if we can move back to your forked client to upstream.

kangwoo · 2020-07-29T11:37:00Z

Ok. I'll start working.

cstyan

Thanks for updating this. Unfortunately I don't know enough about Eureka to do a full review.

cstyan · 2020-07-31T17:34:31Z

discovery/eureka/metadata_marshaller.go

@@ -0,0 +1,112 @@
+// Copyright 2016 The Prometheus Authors


nit, the copyright year for files you're adding should be 2020

I will change it.

cstyan · 2020-07-31T17:35:48Z

discovery/eureka/client_test.go

+	defer ts.Close()
+
+	apps, err := fetchApps(context.TODO(), []string{ts.URL}, &http.Client{})
+	if err != nil {


you can use testutil.Ok(t, err) to fail if an error was returned

I'll change it to testutil.

cstyan · 2020-07-31T17:37:27Z

discovery/eureka/eureka_test.go

+		}
+	)
+	tgs, err := testUpdateServices(client)
+	if err != errTesting {


can use testutil package in this file

cstyan · 2020-07-31T17:38:48Z

docs/configuration/configuration.md

+[ refresh_interval: <duration> | default = 30s ]
+```
+
+By default every app listed in Marathon will be scraped by Prometheus. If not all


marathon -> eureka?

roidelapluie · 2020-07-31T17:41:28Z

Could you squash all the commits into one, so we start fresh?

roidelapluie

I have added some initial comments.

discovery/eureka/eureka.go

roidelapluie · 2020-07-31T19:54:11Z

discovery/eureka/eureka.go

+	appInstanceStatusLabel           = metaLabelPrefix + "app_instance_status"
+	appInstanceIDLabel               = metaLabelPrefix + "app_instance_id"
+	appInstanceMetadataPrefix        = metaLabelPrefix + "app_instance_metadata_"
+	appInstanceMetadataPresentPrefix = metaLabelPrefix + "app_instance_metadatapresent_"


can metadata be present and empty?

I don't think I need it. I will delete it.

roidelapluie · 2020-07-31T19:55:28Z

discovery/eureka/client.go

+const appListPath string = "/apps"
+
+func fetchApps(ctx context.Context, servers []string, client *http.Client) (*Applications, error) {
+	url := randomAppsURL(servers)


That seems a very strange pattern. I would just limit that to ONE server and let the user put load balancing in front if needed.

I agree, but generally Eureka customers implement and use client load balancing devices. I'll think about a better way.

Is that really how the java client works?

Is that really how the java client works?

If the request fails, change the server address and try again.

https://github.com/Netflix/eureka/blob/master/eureka-client/src/main/java/com/netflix/discovery/shared/transport/decorator/RetryableEurekaHttpClient.java#L98

I guess that the users who would want that in prometheus could just have multiple eureka_sd_configs in the same job, with one server each,

I will change it to limit it to ONE server.

roidelapluie · 2020-07-31T19:56:27Z

discovery/eureka/client.go

+		resp.Body.Close()
+	}()
+
+	if (resp.StatusCode < 200) || (resp.StatusCode >= 300) {


resp.StatusCode / 100 == 2

roidelapluie · 2020-07-31T19:57:06Z

discovery/eureka/client.go

+	}()
+
+	if (resp.StatusCode < 200) || (resp.StatusCode >= 300) {
+		return nil, errors.Errorf("non 2xx status '%v' response during eureka service discovery", resp.StatusCode)


Be more specific that %v, e.g. %d.

roidelapluie · 2020-07-31T20:00:03Z

discovery/eureka/eureka.go

+	}
+
+	// Remove services which did disappear.
+	for source := range d.lastRefresh {


Why is that needed?

If the application restarts, it may get out of Discovery.
To avoid immediate removal from the monitoring target if removed from the Discovery.

This is not a correct behavior from the Prometheus perspective. Can you remove this, + lastRefresh?

ok. I will remove it

roidelapluie · 2020-07-31T20:01:13Z

discovery/eureka/metadata_marshaller.go

+	Class   string `xml:"class,attr" json:"@class"`
+}
+
+func (s *MetaData) MarshalXML(e *xml.Encoder, start xml.StartElement) error {


Why would we need to marshal to xml?

We don't need a marshal. I will delete it

roidelapluie · 2020-07-31T20:01:43Z

discovery/eureka/metadata_marshaller.go

+	return nil
+}
+
+func (s *MetaData) MarshalJSON() ([]byte, error) {


I think we need XML or json, not both,

I will delete json.

kangwoo · 2020-07-31T20:33:30Z

Add Eureka Service Discovery

A example scrape configuration for running Prometheus on a Eureka

scrape_configs:

  # Discover Eureka services to scrape.
  - job_name: 'eureka'

    # Scrape Eureka itself to discover new services every minute.
    eureka_sd_configs:
      - servers:
          - http://localhost:8761/eureka
        refresh_interval: 60s

Metdata

You can use Eureka's application instance metadata.

If you are using SpringBoot, you can add metadata using eureka.instance.metadataMap like this:

application.yaml (spring-boot)

eureka:
instance:
  metadataMap:
    "prometheus.scrape": "true"
    "prometheus.path": "/actuator/prometheus"
    "prometheus.port": "8080"

Example relabel to scrape only application that have "prometheus.scrape = true" metadata.

prometheus.xml

- source_labels: [__meta_eureka_app_instance_metadata_prometheus_scrape]
  action: keep
  regex: true

application.yaml (spring-boot)

eureka:
instance:
  metadataMap:
    "prometheus.scrape": "true"

Example relabel to customize metric path based on application "prometheus.path = " annotation.

prometheus.yaml

- source_labels: [__meta_eureka_app_instance_metadata_prometheus_path]
  action: replace
  target_label: __metrics_path__
  regex: (.+)

application.yaml (spring-boot)

eureka:
instance:
  metadataMap:
    "prometheus.path": "/actuator/prometheus"

Example relabel to scrape only single, desired port for the application based on application "prometheus.port = " metadata.

prometheus.xml

- source_labels: [__address__, __meta_eureka_app_instance_metadata_prometheus_port]
  action: replace
  regex: ([^:]+)(?::\d+)?;(\d+)
  replacement: $1:$2
  target_label: __address__

application.yaml (spring-boot)

eureka:
instance:
  metadataMap:
    "prometheus.port": "8080"

roidelapluie · 2020-07-31T22:25:27Z

Why did you close this?