Skip to content

Istio sidecar injection failing with error - MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found #22463

Closed
@sonujose

Description

@sonujose

Bug Description
Istiod is not responding to new namespaces. Created a new namespace and labelled istio-injectio=enabled. I deployed one app to the namespace and the pod is stuck in init stage. Istio proxy is not able to mount configmap "istio-ca-root-cert"

Unable to mount volumes for pod "azure-vote-front-5bc759676c-7hg5t_am-dev(a45f7c3a-d546-4461-af15-c5442ae39de9)": timeout expired waiting for volumes to attach or mount for pod "am-dev"/"azure-vote-front-5bc759676c-7hg5t".

Last log from istiod -

2020-03-25T10:04:33.132573Z warn k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 2764317 (2764517)

list of unmounted volumes=[istiod-ca-cert]. list of unattached volumes=[default-token-58r9k istio-envoy podinfo istio-token istiod-ca-cert]

Warning FailedMount 3m57s (x33 over 54m) kubelet, aks-linuxpool02-21909056-vmss000000 MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found

[ x] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[x ] Developer Infrastructure

New namespace should have configmap "istio-ca-root-cert" and should deploy the app

Steps to reproduce the bug
This happened intermittently. In one of the namespace I was able to attach the proxy and once the problem started, further no namespace is working.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
Istioctl - client version: 1.5.0
kubectl - server: v1.15.7, client: v1.17.0

How was Istio installed?
istioctl

Environment where bug was observed (cloud vendor, OS, etc)
Azure Kubernetes Engine

Activity

mani0070

mani0070 commented on Mar 29, 2020

@mani0070

I'm facing the same issue. I have delete and recreated the namespace still having the same problem.

lei-tang

lei-tang commented on Mar 30, 2020

@lei-tang
Contributor

@sonujose @mani0070 To reproduce the problem, can you share detailed instructions (e.g., the commands you use) of how you install Istio and how you deploy an example application? Meanwhile, besides the Azure Kubernetes Engine platform mentioned in the issue, do you also notice this problem on other platforms?

qwazerty

qwazerty commented on Mar 30, 2020

@qwazerty

I have the same issue aswell on Azure Kubernetes Engine (v 1.15.7).

# istioctl version
client version: 1.5.0
control plane version: 1.5.0
data plane version: 1.5.0 (17 proxies)

Same log in istiod.
2020-03-30T12:46:54.099810Z warn k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 3142588 (3143468)

Created the namespace with these commands, istio does not seems to add the configmap istio-ca-root-cert in this namespace. Other namespaces are working as intended.

# kubectl create namespace keycloak
namespace/keycloak created
# kubectl label namespace keycloak istio-injection=enabled
namespace/keycloak labeled
# kubectl describe namespaces keycloak
Name:         keycloak
Labels:       istio-injection=enabled
Annotations:  <none>
Status:       Active

No resource quota.

No LimitRange resource.
# kubectl get configmaps -n keycloak
No resources found in keycloak namespace.
# kubectl get configmaps -n default
NAME                 DATA   AGE
istio-ca-root-cert   1      10d
sonujose

sonujose commented on Mar 30, 2020

@sonujose
Author

@lei-tang Deployed Istio using the istioctl tool. Istioctl version (1.5.0) and Kubernetes cluster version (v1.15.7)

 istioctl manifest apply \
     --set values.grafana.enabled=true \
     --set values.prometheus.enabled=true \
     --set values.kiali.enabled=true --set values.kiali.createDemoSecret=true \
     --set values.global.proxy.accessLogFile="/dev/dtdout"

Also deployed the same Istio1.5 in two different aks clusters, and this issue was observed in only one of the cluster. The whole problem started after istiod logged this warning - k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 3142588

Istiod pod restart fixed the issue
I was able to fix this issue by deleting the istiod pod. On restart everything started working fine, till now no problems. After the istiod restart the configmap was automatically created for all new namespaces.

linsun

linsun commented on Mar 30, 2020

@linsun
Member

I'm hitting this issue as well... created 2 random namespaces, and none of them has any configmap created.

linsun

linsun commented on Mar 30, 2020

@linsun
Member
GregHanson

GregHanson commented on Mar 30, 2020

@GregHanson
Member

I found this one when I pulled the 1.5 release branch on 3/27. I went back to this commit and the ConfigMap was getting created again.

howardjohn

howardjohn commented on Mar 30, 2020

@howardjohn
Member

@GregHanson that PR seems entirely unrelated. a restart of istiod also fixes that, do you think it was just "fixed" by a restart not that PR?

linsun

linsun commented on Mar 30, 2020

@linsun
Member

confirmed restart of istiod has helped populated istio-ca-root-cert cm for my non-default namespaces.

GregHanson

GregHanson commented on Mar 30, 2020

@GregHanson
Member

@howardjohn - I didn't mean this commit was the culprit - just that it must have come in to the release branch sometime after it. I hit the problem on Friday when I pulled, reverted to that commit, deleted and recreated my namespaces, and I wasn't able to reproduce

127 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @myidpt@haf@foxracle@dosten@emedina

    Issue actions

      Istio sidecar injection failing with error - MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found · Issue #22463 · istio/istio