Skip to content

Commit

Permalink
Support for scraping multiple mysqld hosts (#651)
Browse files Browse the repository at this point in the history
Add multi-target exporter scrape support.

Breaking change:
`DATA_SOURCE_NAME` is now removed. Local configuration is now supplied by `MYSQLD_EXPORTER_PASSWORD` and command line arguments.

Signed-off-by: Mattias Ängehov <mattias.angehov@castoredc.com>
  • Loading branch information
MattiasAng committed Sep 1, 2022
1 parent c7ab579 commit 593b009
Show file tree
Hide file tree
Showing 13 changed files with 640 additions and 234 deletions.
5 changes: 3 additions & 2 deletions Makefile
Expand Up @@ -22,8 +22,9 @@ STATICCHECK_IGNORE =

DOCKER_IMAGE_NAME ?= mysqld-exporter

test-docker:
@echo ">> testing docker image"
.PHONY: test-docker-single-exporter
test-docker-single-exporter:
@echo ">> testing docker image for single exporter"
./test_image.sh "$(DOCKER_IMAGE_NAME):$(DOCKER_IMAGE_TAG)" 9104

.PHONY: test-docker
65 changes: 51 additions & 14 deletions README.md
Expand Up @@ -30,15 +30,50 @@ NOTE: It is recommended to set a max connection limit for the user to avoid over

### Running

Running using an environment variable:

export DATA_SOURCE_NAME='user:password@(hostname:3306)/'
./mysqld_exporter <flags>
##### Single exporter mode

Running using ~/.my.cnf:

./mysqld_exporter <flags>

##### Multi-target support

This exporter supports the multi-target pattern. This allows running a single instance of this exporter for multiple MySQL targets.

To use the multi-target functionality, send an http request to the endpoint /probe?target=foo:5432 where target is set to the DSN of the MySQL instance to scrape metrics from.

To avoid putting sensitive information like username and password in the URL, you can have multiple configurations in `config.my-cnf` file and match it by adding `&auth_module=<section>` to the request.

Sample config file for multiple configurations

[client]
user = foo
password = foo123
[client.servers]
user = bar
password = bar123

On the prometheus side you can set a scrape config as follows

- job_name: mysql # To get metrics about the mysql exporter’s targets
params:
# Not required. Will match value to child in config file. Default value is `client`.
auth_module: client.servers
static_configs:
- targets:
# All mysql hostnames to monitor.
- server1:3306
- server2:3306
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
# The mysqld_exporter host:port
replacement: localhost:9104

##### Flag format
Example format for flags for version > 0.10.0:

--collect.auto_increment.columns
Expand Down Expand Up @@ -102,6 +137,8 @@ collect.heartbeat.utc | 5.1 | U
### General Flags
Name | Description
-------------------------------------------|--------------------------------------------------------------------------------------------------
mysqld.address | Hostname and port used for connecting to MySQL server, format: `host:port`. (default: `locahost:3306`)
mysqld.username | Username to be used for connecting to MySQL Server
config.my-cnf | Path to .my.cnf file to read MySQL credentials from. (default: `~/.my.cnf`)
log.level | Logging verbosity (default: info)
exporter.lock_wait_timeout | Set a lock_wait_timeout (in seconds) on the connection to avoid long metadata locking. (default: 2)
Expand All @@ -112,6 +149,15 @@ web.listen-address | Address to listen on for web interf
web.telemetry-path | Path under which to expose metrics.
version | Print the version information.

### Environment Variables
Name | Description
-------------------------------------------|--------------------------------------------------------------------------------------------------
MYSQLD_EXPORTER_PASSWORD | Password to be used for connecting to MySQL Server

### Configuration precedence

If you have configured cli with both `mysqld` flags and a valid configuration file, the options in the configuration file will override the flags for `client` section.

## TLS and basic authentication

The MySQLd Exporter supports TLS and basic authentication.
Expand All @@ -120,12 +166,6 @@ To use TLS and/or basic authentication, you need to pass a configuration file
using the `--web.config.file` parameter. The format of the file is described
[in the exporter-toolkit repository](https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md).

### Setting the MySQL server's data source name

The MySQL server's [data source name](http://en.wikipedia.org/wiki/Data_source_name)
must be set via the `DATA_SOURCE_NAME` environment variable.
The format of this variable is described at https://github.com/go-sql-driver/mysql#dsn-data-source-name.

## Customizing Configuration for a SSL Connection

If The MySQL server supports SSL, you may need to specify a CA truststore to verify the server's chain-of-trust. You may also need to specify a SSL keypair for the client side of the SSL connection. To configure the mysqld exporter to use a custom CA certificate, add the following to the mysql cnf file:
Expand All @@ -141,8 +181,6 @@ ssl-key=/path/to/ssl/client/key
ssl-cert=/path/to/ssl/client/cert
```

Customizing the SSL configuration is only supported in the mysql cnf file and is not supported if you set the mysql server's data source name in the environment variable DATA_SOURCE_NAME.


## Using Docker

Expand All @@ -157,9 +195,8 @@ docker pull prom/mysqld-exporter
docker run -d \
-p 9104:9104 \
--network my-mysql-network \
-e DATA_SOURCE_NAME="user:password@(hostname:3306)/" \
prom/mysqld-exporter
```
--config.my-cnf=<path_to_cnf>

## heartbeat

Expand Down
229 changes: 229 additions & 0 deletions config/config.go
@@ -0,0 +1,229 @@
// Copyright 2022 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package config

import (
"crypto/tls"
"crypto/x509"
"fmt"
"net"
"os"
"sync"

"github.com/go-kit/log"
"github.com/go-kit/log/level"

"github.com/go-sql-driver/mysql"
"github.com/prometheus/client_golang/prometheus"

"gopkg.in/ini.v1"
)

var (
configReloadSuccess = prometheus.NewGauge(prometheus.GaugeOpts{
Namespace: "mysqld_exporter",
Name: "config_last_reload_successful",
Help: "Mysqld exporter config loaded successfully.",
})

configReloadSeconds = prometheus.NewGauge(prometheus.GaugeOpts{
Namespace: "mysqld_exporter",
Name: "config_last_reload_success_timestamp_seconds",
Help: "Timestamp of the last successful configuration reload.",
})

cfg *ini.File

opts = ini.LoadOptions{
// Do not error on nonexistent file to allow empty string as filename input
Loose: true,
// MySQL ini file can have boolean keys.
AllowBooleanKeys: true,
}

err error
)

type Config struct {
Sections map[string]MySqlConfig
}

type MySqlConfig struct {
User string `ini:"user"`
Password string `ini:"password"`
Host string `ini:"host"`
Port int `ini:"port"`
Socket string `ini:"socket"`
SslCa string `ini:"ssl-ca"`
SslCert string `ini:"ssl-cert"`
SslKey string `ini:"ssl-key"`
TlsInsecureSkipVerify bool `ini:"ssl-skip-verfication"`
}

type MySqlConfigHandler struct {
sync.RWMutex
TlsInsecureSkipVerify bool
Config *Config
}

func (ch *MySqlConfigHandler) GetConfig() *Config {
ch.RLock()
defer ch.RUnlock()
return ch.Config
}

func (ch *MySqlConfigHandler) ReloadConfig(filename string, mysqldAddress string, mysqldUser string, tlsInsecureSkipVerify bool, logger log.Logger) error {
var host, port string
defer func() {
if err != nil {
configReloadSuccess.Set(0)
} else {
configReloadSuccess.Set(1)
configReloadSeconds.SetToCurrentTime()
}
}()

if cfg, err = ini.LoadSources(
opts,
[]byte("[client]\npassword = ${MYSQLD_EXPORTER_PASSWORD}\n"),
filename,
); err != nil {
return fmt.Errorf("failed to load %s: %w", filename, err)
}

if host, port, err = net.SplitHostPort(mysqldAddress); err != nil {
return fmt.Errorf("failed to parse address: %w", err)
}

if clientSection := cfg.Section("client"); clientSection != nil {
if cfgHost := clientSection.Key("host"); cfgHost.String() == "" {
cfgHost.SetValue(host)
}
if cfgPort := clientSection.Key("port"); cfgPort.String() == "" {
cfgPort.SetValue(port)
}
if cfgUser := clientSection.Key("user"); cfgUser.String() == "" {
cfgUser.SetValue(mysqldUser)
}
}

cfg.ValueMapper = os.ExpandEnv
config := &Config{}
m := make(map[string]MySqlConfig)
for _, sec := range cfg.Sections() {
sectionName := sec.Name()

if sectionName == "DEFAULT" {
continue
}

mysqlcfg := &MySqlConfig{
TlsInsecureSkipVerify: tlsInsecureSkipVerify,
}
if err != nil {
level.Error(logger).Log("msg", "failed to load config", "section", sectionName, "err", err)
continue
}

err = sec.StrictMapTo(mysqlcfg)
if err != nil {
level.Error(logger).Log("msg", "failed to parse config", "section", sectionName, "err", err)
continue
}
if err := mysqlcfg.validateConfig(); err != nil {
level.Error(logger).Log("msg", "failed to validate config", "section", sectionName, "err", err)
continue
}

m[sectionName] = *mysqlcfg
}
config.Sections = m
if len(config.Sections) == 0 {
return fmt.Errorf("no configuration found")
}
ch.Lock()
ch.Config = config
ch.Unlock()
return nil
}

func (m MySqlConfig) validateConfig() error {
if m.User == "" {
return fmt.Errorf("no user specified in section or parent")
}
if m.Password == "" {
return fmt.Errorf("no password specified in section or parent")
}

return nil
}

func (m MySqlConfig) FormDSN(target string) (string, error) {
var dsn, host, port string

user := m.User
password := m.Password
if target == "" {
host := m.Host
port := m.Port
socket := m.Socket
if socket != "" {
dsn = fmt.Sprintf("%s:%s@unix(%s)/", user, password, socket)
} else {
dsn = fmt.Sprintf("%s:%s@tcp(%s:%d)/", user, password, host, port)
}
} else {
if host, port, err = net.SplitHostPort(target); err != nil {
return dsn, fmt.Errorf("failed to parse target: %s", err)
}
dsn = fmt.Sprintf("%s:%s@tcp(%s:%s)/", user, password, host, port)
}

if m.SslCa != "" {
if err := m.CustomizeTLS(); err != nil {
err = fmt.Errorf("failed to register a custom TLS configuration for mysql dsn: %w", err)
return dsn, err
}
dsn = fmt.Sprintf("%s?tls=custom", dsn)
}

return dsn, nil
}

func (m MySqlConfig) CustomizeTLS() error {
var tlsCfg tls.Config
caBundle := x509.NewCertPool()
pemCA, err := os.ReadFile(m.SslCa)
if err != nil {
return err
}
if ok := caBundle.AppendCertsFromPEM(pemCA); ok {
tlsCfg.RootCAs = caBundle
} else {
return fmt.Errorf("failed parse pem-encoded CA certificates from %s", m.SslCa)
}
if m.SslCert != "" && m.SslKey != "" {
certPairs := make([]tls.Certificate, 0, 1)
keypair, err := tls.LoadX509KeyPair(m.SslCert, m.SslKey)
if err != nil {
return fmt.Errorf("failed to parse pem-encoded SSL cert %s or SSL key %s: %w",
m.SslCert, m.SslKey, err)
}
certPairs = append(certPairs, keypair)
tlsCfg.Certificates = certPairs
}
tlsCfg.InsecureSkipVerify = m.TlsInsecureSkipVerify
mysql.RegisterTLSConfig("custom", &tlsCfg)
return nil
}

0 comments on commit 593b009

Please sign in to comment.