Description
Using Kubernetes v1.20.0
Attempting to create pvc's, they all remain in "pending" status
Doing a kubectl logs nfs-client-provisioner gives this:
I1210 14:42:01.396466 1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs
I1210 14:42:01.396534 1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.496922 1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.497152 1 controller.go:987] provision "default/pvc1" class "managed-nfs-storage": started
I1210 14:42:01.497157 1 controller.go:987] provision "default/test-claim" class "managed-nfs-storage": started
E1210 14:42:01.500487 1 controller.go:1004] provision "default/pvc1" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
E1210 14:42:01.500502 1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
selfLink has been disabled in v1.20.0
kubernetes/enhancements#1164
Activity
fangedhex commentedon Dec 10, 2020
I can confirm that I have this problem too on my new cluster using Kubernetes v1.20 with those same logs/errors when creating a test pvc.
nealman13 commentedon Dec 10, 2020
Current workaround is to edit /etc/kubernetes/manifests/kube-apiserver.yaml
Under here:
Add this line:
- --feature-gates=RemoveSelfLink=false
The do this:
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
(I had to do it twice to get it to work)
petermicuch commentedon Dec 10, 2020
Yep, exactly the same issue. In k8s 1.20 they have removed selfLink as per release notes.
Adding selfLink manually does not help, as that is removed by API server immediatelly.
fangedhex commentedon Dec 10, 2020
I used @nealman13's workaround until a fix is made and it works now. Thanks !
petermicuch commentedon Dec 10, 2020
Just to add to what @nealman13 provided. Before first leader election log, there is this error:
petermicuch commentedon Dec 10, 2020
I think the problem is in using the old client-go by this replace, but I am not 100% sure if that is the only legacy we pull here.
Same problem was discussed in #94660 and this was suggested solution.
james-d-elliott commentedon Dec 13, 2020
Just a note about
/etc/kubernetes/manifests/kube-apiserver.yaml
. When running the control-plane as an in-cluster service you should not need to apply changes to this file ever (via kubectl apply). As soon as you save the change to disk the apiserver pods will detect it and should conduct a Rolling Update. You can watch the progress withwatch kubectl get pods --namespace kube-system
. You can confirm the behavior after deleting the provisioner pod and letting it restart.selfLink
lensapp/lens#1767iming0319 commentedon Dec 18, 2020
The workaround will no longer work after 1.20.
"selfLink is a URL representing this object. Populated by the system. Read-only. DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release."
degraded version of k8s because of unfixed issue
75 remaining items