I'm facing the same issue. I have delete and recreated the namespace still having the same problem.

@sonujose @mani0070 To reproduce the problem, can you share detailed instructions (e.g., the commands you use) of how you install Istio and how you deploy an example application? Meanwhile, besides the Azure Kubernetes Engine platform mentioned in the issue, do you also notice this problem on other platforms?

I have the same issue aswell on Azure Kubernetes Engine (v 1.15.7).

# istioctl version
client version: 1.5.0
control plane version: 1.5.0
data plane version: 1.5.0 (17 proxies)

Same log in istiod.
2020-03-30T12:46:54.099810Z warn k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 3142588 (3143468)

Created the namespace with these commands, istio does not seems to add the configmap istio-ca-root-cert in this namespace. Other namespaces are working as intended.

# kubectl create namespace keycloak
namespace/keycloak created
# kubectl label namespace keycloak istio-injection=enabled
namespace/keycloak labeled
# kubectl describe namespaces keycloak
Name:         keycloak
Labels:       istio-injection=enabled
Annotations:  <none>
Status:       Active

No resource quota.

No LimitRange resource.
# kubectl get configmaps -n keycloak
No resources found in keycloak namespace.
# kubectl get configmaps -n default
NAME                 DATA   AGE
istio-ca-root-cert   1      10d

@lei-tang Deployed Istio using the istioctl tool. Istioctl version (1.5.0) and Kubernetes cluster version (v1.15.7)

 istioctl manifest apply \
     --set values.grafana.enabled=true \
     --set values.prometheus.enabled=true \
     --set values.kiali.enabled=true --set values.kiali.createDemoSecret=true \
     --set values.global.proxy.accessLogFile="/dev/dtdout"

Also deployed the same Istio1.5 in two different aks clusters, and this issue was observed in only one of the cluster. The whole problem started after istiod logged this warning - k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: watch of *v1.ConfigMap ended with: too old resource version: 3142588

Istiod pod restart fixed the issue
I was able to fix this issue by deleting the istiod pod. On restart everything started working fine, till now no problems. After the istiod restart the configmap was automatically created for all new namespaces.

I'm hitting this issue as well... created 2 random namespaces, and none of them has any configmap created.

istiod.log

cc @howardjohn

I found this one when I pulled the 1.5 release branch on 3/27. I went back to this commit and the ConfigMap was getting created again.

@GregHanson that PR seems entirely unrelated. a restart of istiod also fixes that, do you think it was just "fixed" by a restart not that PR?

confirmed restart of istiod has helped populated istio-ca-root-cert cm for my non-default namespaces.

@howardjohn - I didn't mean this commit was the culprit - just that it must have come in to the release branch sometime after it. I hit the problem on Friday when I pulled, reverted to that commit, deleted and recreated my namespaces, and I wasn't able to reproduce

@lei-tang I have installed istio using

apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
spec:
  # Use the default profile as the base
  # More details at: https://istio.io/docs/setup/additional-setup/config-profiles/
  profile: default
  values:
    global:
      # Ensure that the Istio pods are only scheduled to run on Linux nodes
      defaultNodeSelector:
        beta.kubernetes.io/os: linux
      # Enable mutual TLS for the control plane
      controlPlaneSecurityEnabled: true
      mtls:
        # Require all service to service communication to have mtls
        enabled: false
    grafana:
      # Enable Grafana deployment for analytics and monitoring dashboards
      enabled: true
      security:
        # Enable authentication for Grafana
        enabled: true
    kiali:
      # Enable the Kiali deployment for a service mesh observability dashboard
      enabled: true
    tracing:
      # Enable the Jaeger deployment for tracing
      enabled: true

run the below command : istio-1.5.1

istioctl manifest apply -f istio_controlplane_config.yaml 

The name space is labelled with
Labels: istio-injection=enabled
and deployed the application using helm. when the label is removed I can deploy the application.

Below is the error message
6m25s Warning FailedMount pod/vote-register-dev-78588cfcf6-kc568 MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
6m17s Warning FailedMount pod/vote-register-dev-78588cfcf6-kc568 Unable to mount volumes for pod "vote-register-dev-78588cfcf6-kc568_vote-dev(f4b4b9b1-766b-428c-a970-62a59c497a66)": timeout expired waiting for volumes to attach or mount for pod "vote-dev"/"vote-register-dev-78588cfcf6-kc568". list of unmounted volumes=[istiod-ca-cert]. list of unattached volumes=[default-token-brkz8 istio-envoy podinfo istio-token istiod-ca-cert]

too old resource version: 3142588

Which means the resource of the cluster updates frequently before push all updates to the client side.

When fails with this, the reflector will re list-watch again.

@howardjohn It should not be related to resync period.

BTW, in recent k8s, a feature called watchBookMark is introduced to mitigate too old resource version

The current implementation of the namespace controller is complicated and difficult to reason its correctness. The PR #22613 is created to simplify the implementation and may solve the bugs in this issue. With #22613, the implementation becomes very simple:

• The namespace controller only handles the events of adding a namespace and updating a namespace. The event handler simply writes the configmap to a namespace. The delete events are not handled because when the namespace is deleted, the configmap in the namespace is deleted too and there is no extra actions needed. The controller for configmap is removed to simplify the implementation.
• The update handler of the namespace controller periodically writes the configmap to a namespace, which is useful when the namespace controller needs to sync the configmaps in namespaces. If only the adding events are handled, the namespace controller may miss the adding events and cause the configmap to be out-of-sync.

I have been facing this issue for a while. It was happening very regularly for me before. Moving from 2 to 1 istiod replica seems to have helped quite a bit. A restart of istiod always seems to resolve the issue.

I was facing the same problem, in version 1.5.4, but when restart the istiod restart fixed the problem. before restart even though multiple namespaces were created, configmap was not getting instantiated for the namespaces.

@rajivml @Mdirfan6 Actually this is fixed in 1.5.6, please see the release announcement
https://istio.io/latest/news/releases/1.5.x/announcing-1.5.6/.

Thanks for the update @knight42

Still facing the same issue in 1.5.6

Still facing the same issue in 1.5.6

yes we too observed it on 1.5.6

I also took action with istiod restart.

Istio 1.7.2
Having restarts in istiod, ingressgateway and also failed to mount.
it's really bad, and istiod restart not helps.

Any workaround here?

Same issue in 1.8.0. Looking forward to any workaround here.

meet same issue in 1.8.

meet same issue in 1.8.

meet same issue in 1.8.1

meet same issue in 1.9.0

meet same issue in 1.10.0

Same

@howardjohn

Also getting the same issue.. have deleted the namespace, restarted the pods, killed the pods, carried out the rollback - no luck. Also using AWS EKS, thanks.

same issue following istio-csr getting_started.md

I am also getting this error using EKS 1.21 with Istio 1.11.5. Everything works fine, but this error is causing concern for the developers on our platform.

I was also getting the same error using EKS 1.21 with Istio 1.11.5. Turns out that the error was because I was trying to install istio in kube-system namespace. Installing in another namespace fixed the issue for me. I knew that Istio disabled kube-system proxy for sidecar injection but did not expect it to be a problem for running ingress-gateway pods.

Interestingly, I did not have issues with EKS 1.20 and Istio 1.9.5 even for kube-system namespace. It still works if you upgrade from 1.9.5 to 1.11.5 but a fresh install does not work. I think the issue may be due to Istio 1.11.5 not creating configmap istio-ca-root-cert in kube-system.

Same issue for me on a 4-node RasPi-4 cluster with Istio 1.13.2

Still seeing that kube-system does not have configmap istio-ca-root-cert created in istio 1.12.4. @abhinavgpt @howardjohn, are you aware of any related changes associated with it?

This issue needs to be re-opened. It definitely can still occur in all versions

This was being caused by insufficient CPU and RAM in my eks cluster. I solved by replacing the replicas: 3 and minReplicas:3 in istio.yaml to using single replica. I found the error by examining the istiod pod's logs.

I'm hitting this issue as well... created 2 random namespaces, and none of them has any configmap created.

i seem too