Skip to content

request help: apisix cannot connect to etcd when etcd auth enabled #5608

New issue
New issue
Closed
#5640
Closed
request help: apisix cannot connect to etcd when etcd auth enabled#5608
#5640
@MizuhaHimuraki

Description

@MizuhaHimuraki
MizuhaHimuraki
opened on Nov 25, 2021 · edited by MizuhaHimuraki

Issue description

Apisix cannot work when etcd auth is enabled, but apisix-dashboard does.
Did I miss any key configuration items?

STEP 1: create the etcd user and role

# etcdctl user get apisix
User: apisix
Roles: apisix

# etcdctl  role get apisix
Role apisix
KV Read:
	[/apisix, /apisiy) (prefix /apisix)
KV Write:
	[/apisix, /apisiy) (prefix /apisix)

STEP 2: enable etcd auth

#etcdctl auth enable
Authentication Enabled

STEP 3: create apisix ssl pems

#cat apisix-csr.json
{
  "CN": "apisix",
  "hosts": [
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "DUMMY",
      "L": "DUMMY",
      "O": "etcd",
      "OU": "DUMMY"
    }
  ]
}

STEP 4: make the etcd tls config

# cat conf/config.yaml
nginx_config:
  http:
    lua_shared_dicts:
      s_cache: 8m
  enable_cpu_affinity: false
apisix:
  node_listen: 80
  ssl:
    listen_port: 443
    ssl_trusted_certificate: /etc/pki/tls/certs/ca-bundle.crt
    key_encrypt_salt: -
  allow_admin: [ 127.0.0.0/24, 0.0.0.0/0  ]
  enable_ipv6: false
  admin_key:
    - name: "admin"
      key: -
      role: admin

etcd:
  host: [ https://prod-k8s0.x.com:2379,https://prod-k8s1.x.com:2379,https://prod-k8s2.x.com:2379 ]
  prefix: /apisix-7ac5eb62
  timeout: 10
  health_check_timeout: 60
  ssl_trusted_certificate: /tmp/ca.pem
  tls:
    verify: true
    cert: /tmp/apisix.pem
    key: /tmp/apisix-key.pem

STEP 5 boot apisix

# apisix start
/usr/bin/apisix: line 32: which: command not found
lua /usr/local/apisix/apisix/cli/apisix.lua start
lua_shared_dicts is deprecated, use custom_lua_shared_dict instead
nginx: [error] [lua] init.lua:86: http_init(): failed to load the configuration: failed to read etcd dir

# etcdctl get --prefix /api

STEP 6 try use dashboard with same etcd TLS config create a new route

# cat 
......
  etcd:
    endpoints:
      - https://prod-k8s0.x.com:2379
      - https://prod-k8s1.x.com:2379
      - https://prod-k8s2.x.com:2379
    mtls:
      key_file: " /tmp/apisix-key.pem"
      cert_file: "/tmp/apisix.pem"
      ca_file: "/tmp/ca.pem"
    prefix: /apisix-7ac5eb62
......

# etcdctl get --prefix /api
/apisix-7ac5eb62/routes/383017729642202494
{"id":"383017729642202494","create_time":1637825957,"update_time":1637825957,"uri":"/*","name":"1","methods":["GET","POST","PUT","DELETE","PATCH","HEAD","OPTIONS","CONNECT","TRACE"],"host":"test.x.om","upstream":{"nodes":{"127.0.0.1:9000":1},"timeout":{"connect":6,"send":6,"read":6},"type":"roundrobin","scheme":"http","pass_host":"pass","keepalive_pool":{"idle_timeout":60,"requests":1000,"size":320}},"labels":{"API_VERSION":"1"},"status":1}

STEP 7 disable etcd auth and reboot apisix

# etcdctl get --prefix /api
/apisix-7ac5eb62/consumers/
init_dir
/apisix-7ac5eb62/global_rules/
init_dir
/apisix-7ac5eb62/plugin_configs/
init_dir
/apisix-7ac5eb62/plugin_metadata/
init_dir
/apisix-7ac5eb62/plugins/
init_dir
/apisix-7ac5eb62/proto/
init_dir
/apisix-7ac5eb62/routes/
init_dir
/apisix-7ac5eb62/services/
init_dir
/apisix-7ac5eb62/ssl/
init_dir
/apisix-7ac5eb62/stream_routes/
init_dir
/apisix-7ac5eb62/upstreams/
init_dir
/apisix-7ac5eb62/routes/383017729642202494
{"id":"383017729642202494","create_time":1637825957,"update_time":1637825957,"uri":"/*","name":"1","methods":["GET","POST","PUT","DELETE","PATCH","HEAD","OPTIONS","CONNECT","TRACE"],"host":"test.x.om","upstream":{"nodes":{"127.0.0.1:9000":1},"timeout":{"connect":6,"send":6,"read":6},"type":"roundrobin","scheme":"http","pass_host":"pass","keepalive_pool":{"idle_timeout":60,"requests":1000,"size":320}},"labels":{"API_VERSION":"1"},"status":1}

Environment

  • apisix version : 2.10.1
  • OS : docker image: centos:7
  • OpenResty / Nginx version : nginx version: openresty/1.19.9.1
  • etcd version: etcd Version: 3.4.3

Activity

tokers

tokers commented on Nov 25, 2021

@tokers
tokers
on Nov 25, 2021
Contributor

It seems that you don't configure the username and password fields in APISIX configuration, so it won't carry the credential to ETCD.

MizuhaHimuraki

MizuhaHimuraki commented on Nov 25, 2021

@MizuhaHimuraki
MizuhaHimuraki
on Nov 25, 2021
ContributorAuthor

It seems that you don't configure the username and password fields in APISIX configuration, so it won't carry the credential to ETCD.

Thanks for your reply. I think username and password are not necessary. If an etcd server is launched with the option --client-cert-auth=true, the field of Common Name (CN) in the client’s TLS cert will be used as an etcd user.

I did not set the username and password on dashboard config, but dashboard works well.

And my ssl pem follows this rule.

STEP 3: create apisix ssl pems

#cat apisix-csr.json
{
  "CN": "apisix",
  "hosts": [
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "DUMMY",
      "L": "DUMMY",
      "O": "etcd",
      "OU": "DUMMY"
    }
  ]
}

refer to https://etcd.io/docs/v3.4/op-guide/authentication/#using-tls-common-name

I force enable the debug mode of init_etcd, and got the following errmsg:

CommonName of client sending a request against gateway will be ignored and not used as expected

The error was ignored since the keyword "error" was missing in the response.
details see apisix and etcd

tokers

tokers commented on Nov 26, 2021

@tokers
tokers
on Nov 26, 2021
Contributor

It seems that you don't configure the username and password fields in APISIX configuration, so it won't carry the credential to ETCD.

Thanks for your reply. I think username and password are not necessary. If an etcd server is launched with the option --client-cert-auth=true, the field of Common Name (CN) in the client’s TLS cert will be used as an etcd user.

I did not set the username and password on dashboard config, but dashboard works well.

And my ssl pem follows this rule.

STEP 3: create apisix ssl pems

#cat apisix-csr.json
{
  "CN": "apisix",
  "hosts": [
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "DUMMY",
      "L": "DUMMY",
      "O": "etcd",
      "OU": "DUMMY"
    }
  ]
}

refer to https://etcd.io/docs/v3.4/op-guide/authentication/#using-tls-common-name

I force enable the debug mode of init_etcd, and got the following errmsg:

CommonName of client sending a request against gateway will be ignored and not used as expected

The error was ignored since the keyword "error" was missing in the response. details see apisix and etcd

Just in the docs you share:

Note that this feature cannot be used with gRPC-proxy and gRPC-gateway. This is because gRPC-proxy terminates TLS from its client so all the clients share a cert of the proxy. gRPC-gateway uses a TLS connection internally for transforming HTTP request to gRPC request so it shares the same limitation

APISIX uses the gRPC gateway to communicate with ETCD.

MizuhaHimuraki

MizuhaHimuraki commented on Nov 29, 2021

@MizuhaHimuraki
MizuhaHimuraki
on Nov 29, 2021 · edited by MizuhaHimuraki
ContributorAuthor

Note that this feature cannot be used with gRPC-proxy and gRPC-gateway. This is because gRPC-proxy terminates TLS from its client so all the clients share a cert of the proxy. gRPC-gateway uses a TLS connection internally for transforming HTTP request to gRPC request so it shares the same limitation

APISIX uses the gRPC gateway to communicate with ETCD.

OK, this is a limitation of the etcd itself. I will add username and password fields in the config file to workaround.

CommonName of client sending a request against gateway will be ignored and not used as expected

I think this error message should be recognized during init_etcd. I spent a lot of time analyzing the failure, because the script did not report any errors.

MizuhaHimuraki
mentioned this on Nov 29, 2021
tokers

tokers commented on Nov 30, 2021

@tokers
tokers
on Nov 30, 2021
Contributor

Note that this feature cannot be used with gRPC-proxy and gRPC-gateway. This is because gRPC-proxy terminates TLS from its client so all the clients share a cert of the proxy. gRPC-gateway uses a TLS connection internally for transforming HTTP request to gRPC request so it shares the same limitation

APISIX uses the gRPC gateway to communicate with ETCD.

OK, this is a limitation of the etcd itself. I will add username and password fields in the config file to workaround.

CommonName of client sending a request against gateway will be ignored and not used as expected

I think this error message should be recognized during init_etcd. I spent a lot of time analyzing the failure, because the script did not report any errors.

PR's welcome! We can enhance from two aspects.

  1. Add some comments for the ETCD section configuration;
  2. Optimize the output from ETCD.
spacewander
closed this as completedin #5640on Nov 30, 2021
tokers
mentioned this on Jun 20, 2022
bzp2010
mentioned this on Feb 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @tokers@MizuhaHimuraki

      Issue actions
        request help: apisix cannot connect to etcd when etcd auth enabled · Issue #5608 · apache/apisix