Skip to content

Using Kubernetes v1.20.0, getting "unexpected error getting claim reference: selfLink was empty, can't make reference" #25

Closed
@nealman13

Description

@nealman13

Using Kubernetes v1.20.0

Attempting to create pvc's, they all remain in "pending" status

Doing a kubectl logs nfs-client-provisioner gives this:

I1210 14:42:01.396466 1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs
I1210 14:42:01.396534 1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.496922 1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.497152 1 controller.go:987] provision "default/pvc1" class "managed-nfs-storage": started
I1210 14:42:01.497157 1 controller.go:987] provision "default/test-claim" class "managed-nfs-storage": started
E1210 14:42:01.500487 1 controller.go:1004] provision "default/pvc1" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
E1210 14:42:01.500502 1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

selfLink has been disabled in v1.20.0
kubernetes/enhancements#1164

Activity

fangedhex

fangedhex commented on Dec 10, 2020

@fangedhex

I can confirm that I have this problem too on my new cluster using Kubernetes v1.20 with those same logs/errors when creating a test pvc.

nealman13

nealman13 commented on Dec 10, 2020

@nealman13
Author

Current workaround is to edit /etc/kubernetes/manifests/kube-apiserver.yaml

Under here:

spec:
  containers:
  - command:
    - kube-apiserver

Add this line:
- --feature-gates=RemoveSelfLink=false

The do this:
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

(I had to do it twice to get it to work)

petermicuch

petermicuch commented on Dec 10, 2020

@petermicuch
Contributor

Yep, exactly the same issue. In k8s 1.20 they have removed selfLink as per release notes.

Adding selfLink manually does not help, as that is removed by API server immediatelly.

fangedhex

fangedhex commented on Dec 10, 2020

@fangedhex

I used @nealman13's workaround until a fix is made and it works now. Thanks !

petermicuch

petermicuch commented on Dec 10, 2020

@petermicuch
Contributor

Just to add to what @nealman13 provided. Before first leader election log, there is this error:

E1210 15:41:30.047572       1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nfss", SelfLink:"", UID:"75df6842-e437-4604-9034-2d4d932373be", ResourceVersion:"19431", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63743203519, loc:((nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\"eb-af59-dac986fde3f6\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-10T15:41:30Z\",\"renewTime\":\"2020-12-10T15:41:30Z\",\"leaderTransitions\":1}"}, OwnerReference)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'Le2d51-3afe-11eb-af59-dac986fde3f6 became leader'
I1210 15:41:30.047754       1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs
petermicuch

petermicuch commented on Dec 10, 2020

@petermicuch
Contributor

I think the problem is in using the old client-go by this replace, but I am not 100% sure if that is the only legacy we pull here.

Same problem was discussed in #94660 and this was suggested solution.

james-d-elliott

james-d-elliott commented on Dec 13, 2020

@james-d-elliott

Just a note about /etc/kubernetes/manifests/kube-apiserver.yaml. When running the control-plane as an in-cluster service you should not need to apply changes to this file ever (via kubectl apply). As soon as you save the change to disk the apiserver pods will detect it and should conduct a Rolling Update. You can watch the progress with watch kubectl get pods --namespace kube-system. You can confirm the behavior after deleting the provisioner pod and letting it restart.

iming0319

iming0319 commented on Dec 18, 2020

@iming0319

The workaround will no longer work after 1.20.
"selfLink is a URL representing this object. Populated by the system. Read-only. DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release."

75 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @huntersan9@olcayseker@achernev@jsafrane@abdennour

    Issue actions

      Using Kubernetes v1.20.0, getting "unexpected error getting claim reference: selfLink was empty, can't make reference" · Issue #25 · kubernetes-sigs/nfs-subdir-external-provisioner