Description
Bug Description
Istiod is not responding to new namespaces. Created a new namespace and labelled istio-injectio=enabled. I deployed one app to the namespace and the pod is stuck in init stage. Istio proxy is not able to mount configmap "istio-ca-root-cert"
Unable to mount volumes for pod "azure-vote-front-5bc759676c-7hg5t_am-dev(a45f7c3a-d546-4461-af15-c5442ae39de9)": timeout expired waiting for volumes to attach or mount for pod "am-dev"/"azure-vote-front-5bc759676c-7hg5t".
Last log from istiod -
2020-03-25T10:04:33.132573Z warn k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 2764317 (2764517)
list of unmounted volumes=[istiod-ca-cert]. list of unattached volumes=[default-token-58r9k istio-envoy podinfo istio-token istiod-ca-cert]
Warning FailedMount 3m57s (x33 over 54m) kubelet, aks-linuxpool02-21909056-vmss000000 MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
[ x] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[x ] Developer Infrastructure
New namespace should have configmap "istio-ca-root-cert" and should deploy the app
Steps to reproduce the bug
This happened intermittently. In one of the namespace I was able to attach the proxy and once the problem started, further no namespace is working.
Version (include the output of istioctl version --remote
and kubectl version
and helm version
if you used Helm)
Istioctl - client version: 1.5.0
kubectl - server: v1.15.7, client: v1.17.0
How was Istio installed?
istioctl
Environment where bug was observed (cloud vendor, OS, etc)
Azure Kubernetes Engine
Activity
mani0070 commentedon Mar 29, 2020
I'm facing the same issue. I have delete and recreated the namespace still having the same problem.
lei-tang commentedon Mar 30, 2020
@sonujose @mani0070 To reproduce the problem, can you share detailed instructions (e.g., the commands you use) of how you install Istio and how you deploy an example application? Meanwhile, besides the Azure Kubernetes Engine platform mentioned in the issue, do you also notice this problem on other platforms?
qwazerty commentedon Mar 30, 2020
I have the same issue aswell on Azure Kubernetes Engine (v 1.15.7).
Same log in istiod.
2020-03-30T12:46:54.099810Z warn k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 3142588 (3143468)
Created the namespace with these commands, istio does not seems to add the configmap
istio-ca-root-cert
in this namespace. Other namespaces are working as intended.sonujose commentedon Mar 30, 2020
@lei-tang Deployed Istio using the istioctl tool. Istioctl version (1.5.0) and Kubernetes cluster version (v1.15.7)
Also deployed the same Istio1.5 in two different aks clusters, and this issue was observed in only one of the cluster. The whole problem started after istiod logged this warning -
k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 3142588
Istiod pod restart fixed the issue
I was able to fix this issue by deleting the istiod pod. On restart everything started working fine, till now no problems. After the istiod restart the configmap was automatically created for all new namespaces.
linsun commentedon Mar 30, 2020
I'm hitting this issue as well... created 2 random namespaces, and none of them has any configmap created.
linsun commentedon Mar 30, 2020
istiod.log
cc @howardjohn
GregHanson commentedon Mar 30, 2020
I found this one when I pulled the 1.5 release branch on 3/27. I went back to this commit and the ConfigMap was getting created again.
howardjohn commentedon Mar 30, 2020
@GregHanson that PR seems entirely unrelated. a restart of istiod also fixes that, do you think it was just "fixed" by a restart not that PR?
linsun commentedon Mar 30, 2020
confirmed restart of istiod has helped populated istio-ca-root-cert cm for my non-default namespaces.
GregHanson commentedon Mar 30, 2020
@howardjohn - I didn't mean this commit was the culprit - just that it must have come in to the release branch sometime after it. I hit the problem on Friday when I pulled, reverted to that commit, deleted and recreated my namespaces, and I wasn't able to reproduce
127 remaining items