Skip to content

Failed to change kubernetes CA #67122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shonge opened this issue Aug 8, 2018 · 13 comments
Closed

Failed to change kubernetes CA #67122

shonge opened this issue Aug 8, 2018 · 13 comments
Assignees
Labels
area/kubeadm sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.

Comments

@shonge
Copy link

shonge commented Aug 8, 2018

What happened:
I changed CA cert,CA key and others cert in a existing kubernetes cluster, but I got these errors with kube-apiserver:
E0808 15:47:37.250207 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority]
kubelet error:
Aug 09 00:51:14 k8s01 kubelet[17641]: E0809 00:51:14.323791 17641 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized

kubelet and kube-apiserver are using a same ca.crt.

COMMAND:
Use curl command and new certs got these info
curl -s https://192.168.92.135:6443/api/v1/nodes/ --key /etc/kubernetes/k8s01-key.pem --cert /etc/kubernetes/k8s01.pem --cacert /etc/kubernetes/pki/ca.crt | jq '.items[] .metadata.labels'
{
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/hostname": "k8s01",
"node-role.kubernetes.io/master": ""
}

run api server on master
kube-apiserver --authorization-mode=Node,RBAC --advertise-address=192.168.92.135 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --disable-admission-plugins=Persistent
check-sum api-ca
md5sum /etc/kubernetes/pki/ca.crt
9eb748da44178d5dfa8f3ff2a8ce621a /etc/kubernetes/pki/ca.crt
run kubelet on master
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
check-conf kubelet
cat /var/lib/kubelet/config.yaml | grep ca.crt
clientCAFile: /etc/kubernetes/pki/ca.crt
cat /etc/kubernetes/kubelet.conf | grep ca.crt
certificate-authority: /etc/kubernetes/pki/ca.crt
check-sum kubelet-ca
md5sum /etc/kubernetes/pki/ca.crt
9eb748da44178d5dfa8f3ff2a8ce621a /etc/kubernetes/pki/ca.crt
What you expected to happen:

Anything else we need to know?:
Create cluster commad:
kubeadm init --pod-network-cidr=10.224.0.0/16

Environment:

  • Kubernetes version: v1.11.2
  • OS: CentOS Linux release 7.4.1708
  • Kernel: Linux k8s01 3.10.0-693.el7.x86_64
  • Install tools:
    kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Others:
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Aug 8, 2018
@shonge shonge changed the title Failed to change kubernetes CA Failed to change kubernetes CA sig/auth Aug 8, 2018
@shonge
Copy link
Author

shonge commented Aug 8, 2018

/sig auth

@k8s-ci-robot k8s-ci-robot added sig/auth Categorizes an issue or PR as relevant to SIG Auth. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 8, 2018
@shonge shonge changed the title Failed to change kubernetes CA sig/auth Failed to change kubernetes CA Aug 8, 2018
@hzxuzhonghu
Copy link
Member

Do you notice any heartbeat error between kubelet and kube-apiserever? If heartbeat failed, kubelet will close all connections.

/cc @liggitt

@hzxuzhonghu
Copy link
Member

@liggitt Does kubelet auto reload certificate when it reconnect to kube-apiserver in this case?

@hzxuzhonghu
Copy link
Member

Does kubelet auto reload certificate when it reconnect to kube-apiserver in this case?

Actually yes.

@shonge Do you have multi master?

@shonge
Copy link
Author

shonge commented Aug 8, 2018

kubelet and kube-apiserver on the same server.

@shonge
Copy link
Author

shonge commented Aug 8, 2018

Nope, just one master.

@shonge
Copy link
Author

shonge commented Aug 8, 2018

I restarted kubelet serivce with command:
'service kubelet restart'
'service docker restart' for resart apiserver

@shonge
Copy link
Author

shonge commented Aug 8, 2018

How to reload certificate on kubelet ?

@shonge
Copy link
Author

shonge commented Aug 8, 2018

I changed /var/lib/kubelet/kubeadm-flags.env like this, but nothing happened.
KUBELET_KUBEADM_ARGS=--rotate-certificates --cgroup-driver=systemd --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni

@shonge
Copy link
Author

shonge commented Aug 8, 2018

/area kubeadm

@shonge
Copy link
Author

shonge commented Aug 8, 2018

/sig cluster-lifecycle

@k8s-ci-robot k8s-ci-robot added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label Aug 8, 2018
@liggitt
Copy link
Member

liggitt commented Aug 8, 2018

Restarting the kubelet reloads the certificate from disk. However, if you change the CA, you may need to start back at the bootstrap step. I'm not sure what kubeadm command is used to do that.

@neolit123
Copy link
Member

neolit123 commented Aug 14, 2018

support for certicate rotation is WIP in kubeadm and might make it in 1.12 or in a later version:
kubernetes/kubeadm#206

the kubeadm bootstrap process is separated into phases, but these have an alpha state:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-alpha/

AFAIK, what you have to do right now is bring the cluster down, change the CA and re-create the cluster.

/close

(edit: if you want more feedback you can post in the above issue).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubeadm sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
Projects
None yet
Development

No branches or pull requests

5 participants