Skip to content

Error: failed to create discovery service: failed to create CA: failed to create an istiod CA: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate has expired or is not yet valid: current time 2020-07-03T01:53:13Z is before 2020-07-03T02:00:31Z #25108

Closed
@tanjunchen

Description

@tanjunchen
Member

What do you mean what system do you use?
I think this issue was resolved in 1.6.x

Originally posted by @FrimIdan in #23108 (comment)

The Logs:

kubectl logs -f istiod-788cf6c878-stlg8 -n istio-system

2020-06-30T09:06:56.516278Z	info	grpc: Server.Serve failed to complete security handshake from "10.24.117.128:51858": tls: failed to verify client certificate: x509: certificate signed by unknown authority
2020-06-30T09:07:13.149081Z	info	grpc: Server.Serve failed to complete security handshake from "10.24.117.128:51860": tls: failed to verify client certificate: x509: certificate signed by unknown authority

tail -f /var/log/istio/istio.log

2020-06-30T09:33:59.522059Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc0001186d0, {TRANSIENT_FAILURE connection closed}
2020-06-30T09:33:59.522088Z	info	Channel Connectivity change to TRANSIENT_FAILURE
2020-06-30T09:35:55.178267Z	info	Subchannel Connectivity change to CONNECTING
2020-06-30T09:35:55.178369Z	info	Subchannel picks a new address "istiod.istio-system.svc:32557" to connect
2020-06-30T09:35:55.178643Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc0001186d0, {CONNECTING <nil>}
2020-06-30T09:35:55.178680Z	info	Channel Connectivity change to CONNECTING
2020-06-30T09:35:55.187378Z	info	Subchannel Connectivity change to TRANSIENT_FAILURE
2020-06-30T09:35:55.187417Z	info	pickfirstBalancer: HandleSubConnStateChange: 0xc0001186d0, {TRANSIENT_FAILURE connection closed}
2020-06-30T09:35:55.187430Z	info	Channel Connectivity change to TRANSIENT_FAILURE
2020-06-30T09:35:55.187432Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"

I've followed https://istio.io/docs/examples/virtual-machines/multi-network/ with some changes:

the procedure of deploying test:

  1. kubectl create namespace istio-system
kubectl create secret generic cacerts -n istio-system \
    --from-file=samples/certs/ca-cert.pem \
    --from-file=samples/certs/ca-key.pem \
    --from-file=samples/certs/root-cert.pem \
    --from-file=samples/certs/cert-chain.pem
  1. How was Istio installed?
istioctl install -f manifests/examples/vm/values-istio-meshexpansion-gateways.yaml
  1. the content of cluster.env
ISTIO_SERVICE_CIDR=10.32.0.0/16

ISTIO_INBOUND_PORTS=3306,8080

# 32557 is a port of ingress-gateway NodePort(15012)
ISTIO_PILOT_PORT=32557
  1. CA certificates

the official docs

go run istio.io/istio/security/tools/generate_cert \
      -client -host spiffee://cluster.local/vm/vmname --out-priv key.pem --out-cert cert-chain.pem  -mode citadel

the command to make ca certificate I use . the reason see #23108 (comment)

go run istio.io/istio/security/tools/generate_cert -client -server \ 
-host spiffe://cluster.local/ns/vm/sa/default -signer-cert \ 
samples/certs/ca-cert.pem -signer-priv samples/certs/ca-key.pem \
 --out-priv key.pem --out-cert cert-chain.pem -mode signer

istioctl version --remote

client version: 1.6.0
control plane version: 1.6.0
data plane version: 1.6.0 (9 proxies)

kubectl version

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

the istio cluster os

LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.5.1804 (Core) 
Release:	7.5.1804
Codename:	Core

vm os:

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic

Activity

changed the title [-]mTLS connection from cluster to VM expansion is failing[/-] [+]mTLS connection from cluster to VM expansion is failing certificate: x509: certificate signed by unknown authority[/+] on Jul 1, 2020
clarechu

clarechu commented on Jul 2, 2020

@clarechu

这个应该是你的证书问题了哦!

tanjunchen

tanjunchen commented on Jul 2, 2020

@tanjunchen
MemberAuthor

Yes, certificate issue, how your certificate was generated?

clarechu

clarechu commented on Jul 2, 2020

@clarechu

execution of this command

make -f "${ISTIO_DIR}"/tools/certs/Makefile NAME="${CLUSTER_NAME}" NAMESPACE="${SERVICE_NAMESPACE}" "${CLUSTER_NAME}"-certs-wl
tanjunchen

tanjunchen commented on Jul 2, 2020

@tanjunchen
MemberAuthor

execution of this command

make -f "${ISTIO_DIR}"/tools/certs/Makefile NAME="${CLUSTER_NAME}" NAMESPACE="${SERVICE_NAMESPACE}" "${CLUSTER_NAME}"-certs-wl

sure. i will try this , thanks.

tanjunchen

tanjunchen commented on Jul 2, 2020

@tanjunchen
MemberAuthor

@clarechu what is your istio version? 1.6.0? where is /tools/certs/Makefile? my istio version is 1.6.0

clarechu

clarechu commented on Jul 2, 2020

@clarechu

this is 天坑

$ cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/ca-cert.pem "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/
$ cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/key.pem "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/
$ cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/root-cert.pem "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/
$ cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/`workload-cert-chain.pem` "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"`/cert-chain.pem`

The last file needs to be replaced with name

workload-cert-chain.pem ------> cert-chain.pem

clarechu

clarechu commented on Jul 2, 2020

@clarechu

@clarechu what is your istio version? 1.6.0? where is /tools/certs/Makefile? my istio version is 1.6.0

istio 1.6.0
k8s v1.16.6

tanjunchen

tanjunchen commented on Jul 2, 2020

@tanjunchen
MemberAuthor

@clarechu thanks your reply.

clarechu

clarechu commented on Jul 2, 2020

@clarechu

you're welcome

tanjunchen

tanjunchen commented on Jul 3, 2020

@tanjunchen
MemberAuthor

image

see the document https://istio.io/latest/docs/setup/install/virtual-machine/, but we should pay attention to make -f "${ISTIO_DIR}"/samples/certs/Makefile NAME="${CLUSTER_NAME}" NAMESPACE="${SERVICE_NAMESPACE}" "${CLUSTER_NAME}"-certs-wl. /samples/certs/Makefile The istio branch 1.6.0 source package does not have this Makefile, but the tag 1.6.0 exists in this file. So when using 1.6.0, it is recommended to use curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.0 sh-

@clarechu Thanks for your reply.Thank you very mush.
/close

changed the title [-]mTLS connection from cluster to VM expansion is failing certificate: x509: certificate signed by unknown authority[/-] [+]Error: failed to create discovery service: failed to create CA: failed to create an istiod CA: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate has expired or is not yet valid: current time 2020-07-03T01:53:13Z is before 2020-07-03T02:00:31Z[/+] on Jul 3, 2020
tanjunchen

tanjunchen commented on Jul 3, 2020

@tanjunchen
MemberAuthor

the logs from istiod at the process of installing the istio cluster.

2020-07-03T01:53:13.113287Z	error	failed to create discovery service: failed to create CA: failed to create an istiod CA: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate has expired or is not yet valid: current time 2020-07-03T01:53:13Z is before 2020-07-03T02:00:31Z)
Error: failed to create discovery service: failed to create CA: failed to create an istiod CA: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate has expired or is not yet valid: current time 2020-07-03T01:53:13Z is before 2020-07-03T02:00:31Z

After my testing, I found that the problem is the time zone. The problematic time zone is Fri Jul 3 23:02:24 CST 2020, but Fri Jul 3 08:04:46 PDT 2020 does not cause this problem in another time zone.

2 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @tanjunchen@clarechu@istio-policy-bot

        Issue actions

          Error: failed to create discovery service: failed to create CA: failed to create an istiod CA: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate has expired or is not yet valid: current time 2020-07-03T01:53:13Z is before 2020-07-03T02:00:31Z · Issue #25108 · istio/istio