-
Notifications
You must be signed in to change notification settings - Fork 40.6k
Failed to change kubernetes CA #67122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
sig/auth
/sig auth |
sig/auth
Do you notice any heartbeat error between kubelet and kube-apiserever? If heartbeat failed, kubelet will close all connections. /cc @liggitt |
@liggitt Does kubelet auto reload certificate when it reconnect to kube-apiserver in this case? |
Actually yes. @shonge Do you have multi master? |
kubelet and kube-apiserver on the same server. |
Nope, just one master. |
I restarted kubelet serivce with command: |
How to reload certificate on kubelet ? |
I changed /var/lib/kubelet/kubeadm-flags.env like this, but nothing happened. |
/area kubeadm |
/sig cluster-lifecycle |
Restarting the kubelet reloads the certificate from disk. However, if you change the CA, you may need to start back at the bootstrap step. I'm not sure what kubeadm command is used to do that. |
support for certicate rotation is WIP in kubeadm and might make it in 1.12 or in a later version: the kubeadm bootstrap process is separated into phases, but these have an alpha state: AFAIK, what you have to do right now is bring the cluster down, change the CA and re-create the cluster. /close (edit: if you want more feedback you can post in the above issue). |
Uh oh!
There was an error while loading. Please reload this page.
What happened:
I changed CA cert,CA key and others cert in a existing kubernetes cluster, but I got these errors with kube-apiserver:
E0808 15:47:37.250207 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
kubelet error:
Aug 09 00:51:14 k8s01 kubelet[17641]: E0809 00:51:14.323791 17641 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
kubelet and kube-apiserver are using a same ca.crt.
COMMAND:
Use curl command and new certs got these info
curl -s https://192.168.92.135:6443/api/v1/nodes/ --key /etc/kubernetes/k8s01-key.pem --cert /etc/kubernetes/k8s01.pem --cacert /etc/kubernetes/pki/ca.crt | jq '.items[] .metadata.labels'
{
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/hostname": "k8s01",
"node-role.kubernetes.io/master": ""
}
run api server on master
kube-apiserver --authorization-mode=Node,RBAC --advertise-address=192.168.92.135 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --disable-admission-plugins=Persistent
check-sum api-ca
md5sum /etc/kubernetes/pki/ca.crt
9eb748da44178d5dfa8f3ff2a8ce621a /etc/kubernetes/pki/ca.crt
run kubelet on master
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
check-conf kubelet
cat /var/lib/kubelet/config.yaml | grep ca.crt
clientCAFile: /etc/kubernetes/pki/ca.crt
cat /etc/kubernetes/kubelet.conf | grep ca.crt
certificate-authority: /etc/kubernetes/pki/ca.crt
check-sum kubelet-ca
md5sum /etc/kubernetes/pki/ca.crt
9eb748da44178d5dfa8f3ff2a8ce621a /etc/kubernetes/pki/ca.crt
What you expected to happen:
Anything else we need to know?:
Create cluster commad:
kubeadm init --pod-network-cidr=10.224.0.0/16
Environment:
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
The text was updated successfully, but these errors were encountered: