Skip to content

mTLS connection from cluster to VM expansion is failing #23108

Closed
@FrimIdan

Description

@FrimIdan
Contributor

Bug description
I've tested VM multi network mesh expansion with mTLS connection from the GKE to the VM (EC2) - the connection failed due to:

From the client (sleep-pod) logs:
TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
From the server (VM) logs:
TLS error: 268436504:SSL routines:OPENSSL_internal:TLSV1_ALERT_UNKNOWN_CA

I've followed https://istio.io/docs/examples/virtual-machines/multi-network/ with some changes:

  1. Wanted to caught all inbound and outbound traffic to the VM so I've set ISTIO_SERVICE_CIDR=* and ISTIO_INBOUND_PORTS=*
  2. go run istio.io/istio/security/tools/generate_cert -client -host spiffee://cluster.local/vm/vmname --out-priv key.pem --out-cert cert-chain.pem -mode citadel - command didn't worked because istiod is not self-signed mode so no istio-ca-secret secret. I run the following with spiffe://cluster.local/ns/vm/sa/default - generate_cert -client -host spiffe://cluster.local/ns/vm/sa/default -signer-cert samples/certs/ca-cert.pem -signer-priv samples/certs/ca-key.pem --out-priv key.pem --out-cert cert-chain.pem -mode signer
  3. Due to the fact that the VM is not on the same network as my GKE, I've added an endpoint to the serviceentry (that was created with the istioctl experimental add-to-mesh command) that will direct traffic to the ext-ip of my VM and gave him highest weight then the internal IP
  4. Added a destination rule to encrypt traffic to the VM
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: vm-mtls
spec:
  host: "*.vm.svc.cluster.local"
  trafficPolicy:
    portLevelSettings:
    - port: 
        number: 80
      tls:
        mode: ISTIO_MUTUAL

Expected behavior
mtls connection from the cluster to the vm should work
Steps to reproduce the bug
Follow https://istio.io/docs/examples/virtual-machines/multi-network/ and add a destination rule to use mtls to the VM

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.5.1

How was Istio installed?
istioctl manifest apply
-f install/kubernetes/operator/examples/vm/values-istio-meshexpansion.yaml

Environment where bug was observed (cloud vendor, OS, etc)
GKE, EC2

Activity

FrimIdan

FrimIdan commented on Apr 21, 2020

@FrimIdan
ContributorAuthor

@irisdingbj following our discussion

self-assigned this
on Apr 21, 2020
xichengliudui

xichengliudui commented on Jun 28, 2020

@xichengliudui
Contributor

I encountered the same problem as you, but I did not fix it. what system do you use?

FrimIdan

FrimIdan commented on Jun 28, 2020

@FrimIdan
ContributorAuthor

What do you mean what system do you use?
I think this issue was resolved in 1.6.x

xichengliudui

xichengliudui commented on Jun 29, 2020

@xichengliudui
Contributor

operating system

FrimIdan

FrimIdan commented on Jun 29, 2020

@FrimIdan
ContributorAuthor

I was running on Ubuntu, I think the issue was that the certificates that was generated for the VM wasn't a server side certificates.

tanjunchen

tanjunchen commented on Jun 29, 2020

@tanjunchen
Member

What do you mean what system do you use?
I think this issue was resolved in 1.6.x

I see the same issue in 1.6.0

[root@vm10-master istio-1.6.0]# go run istio.io/istio/security/tools/generate_cert  -client -host spiffee://cluster.local/vm/vmname --out-priv key.pem --out-cert cert-chain.pem  -mode citadel
go: finding istio.io/istio latest
2020-06-29T09:02:34.166049Z	info	Command failed error: exit status 1
, output
Error from server (NotFound): secrets "istio-ca-secret" not found
exit status 1

lsb_release -a

LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.5.1804 (Core) 
Release:	7.5.1804
Codename:	Core
tanjunchen

tanjunchen commented on Jul 3, 2020

@tanjunchen
Member
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @irisdingbj@tanjunchen@FrimIdan@xichengliudui@istio-policy-bot

      Issue actions

        mTLS connection from cluster to VM expansion is failing · Issue #23108 · istio/istio