Closed
Description
Kubernetes 1.5.1 All in One on the baremetal, external ceph cluster.
Using StorageClass and PVC to provision volume.
PVC is stuck in Pending state and log indicates the following error.
Dec 17 09:43:08 kube-1 journal: E1217 14:43:08.506433 1 rbd_util.go:341] rbd: Error creating rbd image: executable file not found in $PATH
Dec 17 09:43:08 kube-1 journal: E1217 14:43:08.506441 1 rbd.go:310] rbd: create volume failed, err: executable file not found in $PATH
RBD executable IS in PATH as you can see from the output below.
[root@kube-1 local-cluster]# which rbd
/usr/bin/rbd
If it is looking for some other executable then question 1: which one exactly? and 2: the error message should specify the name of executable which is missing.
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Activity
thanodnl commentedon Dec 18, 2016
I ran into the same issue and found it strange that kubernetes complained about
rbd
not being on my$PATH
. After digging around I have found why it is complaining about missingrbd
.The process that I have tracked the log line down to is the
kube-controller-manager
. This process is running in its own pod (atleast after installing a cluster withkubeadm
). The container in this pod is missing therbd
binary.I have tried creating a new image from the released
kube-controller-manager
image and addrbd
to it. It was no surprise to find that it didn't work asrbd
probably needs some other shared libraries that I didn't copy over. At leastrbd
was now in my container but non-runnable and that got picked up bykube-controller-manager
.Is there a way to add an other container to the
kube-controller-manager
pod, a container that has a workingrbd
binary in it? Like the container used in https://github.com/ceph/ceph-docker/tree/master/examples/kubernetessbezverk commentedon Dec 18, 2016
Yep, you are right. rbd binary does not exist in kube-controller-manager container.
docker exec -tu root -ti 1dd53ef015ab find / -name rbd
/sys/bus/rbd
/sys/devices/rbd
/sys/module/rbd
/sys/module/libceph/holders/rbd
v1k0d3n commentedon Dec 18, 2016
@thanodnl we had to create a container with kube-controller-manager binary packed in for this purpose. this is a workaround, but you are free to use our image until a true "fix" is in place.
https://quay.io/repository/attcomdev/kube-controller-manager?tab=tags
we've been editing the "image" in /etc/kubernetes/manifests/kube-controller-manager.json post deployment, and that seems to do the trick for us.
let us know if this works for you too?
thanodnl commentedon Dec 18, 2016
I have just started your image and got it to work! Thanks a lot.
Just one question, your image wasn't able to do a lookup of the monitor on its dns name via skyDNS. For now I have put the monitor ip's in the storage class and it works.
Just wondering if you have tried using this image in combination with the ceph monitor running on k8s.
v1k0d3n commentedon Dec 18, 2016
@thanodnl glad it worked (with some tinkering). we have it working just fine as described in our project at: https://github.com/att-comdev/openstack-helm.
if there is anything you find missing, please submit an issue and we can address it. we're at the very beginning of the project, but will be improving it quite a bit over the coming months. ceph is first class in our solution, so these issues need addressed. if you catch something, either submit an issue, or we'd be happy to accept a PR as well. the docker container is listed here: https://github.com/att-comdev/dockerfiles/tree/master/kube-controller-manager. i have plans to look over submitting these 'fixes' in kubernetes proper; i just started looking into it this afternoon.
v1k0d3n commentedon Dec 18, 2016
adding @alanmeadows to the thread (keep an eye on this one).
kongslund commentedon Jan 19, 2017
Thanks, @v1k0d3n. Any progress on a more long term solution?
v1k0d3n commentedon Jan 20, 2017
@kongslund i think the long term goal is to build a custom kube-controller with the pvc tooling you need. i don't want to take over @sbezverk's issue, but I think the reason the executable isn't found because it hasn't been included. we created a kube-controller image with the pvc bins required for ceph. you're welcome to use that, or you may want to create one (based on our example above) for your own needs.
alanmeadows commentedon Jan 20, 2017
@thanodnl What happens is that the k8s controller manager leverages
hostNetwork: true
so it essentially inherits the underlying physical hosts/etc/resolv.conf
. We resolve this today by ensuring all k8s nodes leverage skyDNS as their first resolution server prior to starting kubelet with fallbacks to standard resolution endpoints.The correct solution here for which I'm sure there is an open issue is to support
dnsPolicy: ClusterFirst
on hostNetwork: true hosts. To my knowledge, this has no effect (nor does it error at the API level) when leveraged onhostNetwork
enabled artifacts.At the end of the day, ensuring skyDNS is leveraged from the controller pod allows us to resolve
ceph-mon.ceph
(a k8s deployment of ceph itself) from the controller manager so we can avoid leveraging the monitor IPs directly.4admin2root commentedon Apr 27, 2017
Is there any plan to add ceph rbd to google_containers docker image (e.g, gcr.io/google_containers/kube-controller-manager-amd64) ?
jingxu97 commentedon May 3, 2017
cc @saad-ali
luxas commentedon May 30, 2017
cc @kubernetes/sig-storage-bugs
I guess and think this is out of scope to include rbd, but very unfortunate bug indeed.
jsafrane commentedon May 30, 2017
I am working on kubernetes/enhancements#278 - /usr/bin/rbd (and similar tools) could run in containers. On GKE you would run a daemonset with all Ceph utilities and you won't need anything on the nodes nor master(s).
52 remaining items
dims commentedon May 27, 2018
/close
@sbezverk please reopen if necessary
feresberbeche commentedon May 28, 2018
/reopen
/usr/bin # yum list |grep ceph-common root@kubernetes ceph-common.x86_64 1:10.2.10-0.el7 @Ceph
/usr/bin # ceph -v root@kubernetes ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)
kubectl version root@kubernetes Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-18T23:58:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
I0528 13:39:01.708902 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ceph-pvc", UID:"48891a51-6272-11e8-8ff1-e83935e84dc8", APIVersion:"v1", ResourceVersion:"959049", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "rbd": failed to create rbd image: executable file not found in $PATH, command output:
I still have this problem, I tried to use an external provisioner and I had different problem
I hhave the rdb binary and kubelet binary inside the node @ /usr/bin/
k8s-ci-robot commentedon May 28, 2018
@feresberbeche: you can't re-open an issue/PR unless you authored it or you are assigned to it.
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
shuraa commentedon Dec 22, 2018
@cofyc
Ok, It can work. But what about of several already provisioned PVC/PV by kubernetes.io/rbd provisioner? How can I switch these volumes under the control of external storage controller? What is a clear upgrade path from 1.11 to 1.12?
cofyc commentedon Dec 22, 2018
There is no automatic way, but you can do it manually.
spec.claimRef
filed to point to new claimNo extra action required for RBD provisioning.
r0bj commentedon Jan 7, 2019
External provisioner is great during provisioning volumes but if you want to resize those volumes it's handled by in-tree volume plugin (kube-controller-manager):
So if you want to resize volumes you still need to have rbd binary in kube-controller-manager image.
michaelfig commentedon Feb 3, 2019
@r0bj I solved this by editing the
/etc/kubernetes/manifests/kube-controller-manager.yaml
to use thehyperkube
image.exodussystem commentedon Feb 28, 2019
This is very useful detail explanations.
I installed Kubernetes 1.13.2 with Kubespray
For the first part (Volume Provisioning), kubespray deployment uses /external-storage/rbd-provisioner container. So it works fine. I checked the logs and Ceph created the volume ok
In the second part (Volume Attach/Detach), it failed because rbd binary was not found.
My kubelet is running on CoreOS host (not inside docker/rkt) because and due to to security, kubespray kubelet_deployment_type default is 'host'. I could NOT install ceph-common when setting up these nodes. However, I could invoke 'rbd' binary mounted from ceph/ceph container
Source:
https://stackoverflow.com/questions/37996983/invoking-rbd-docker-from-kubernetes-on-coreos-returns-fork-exec-invalid-argume
I still got the error stating that rbd could not find ceph.conf in these nodes. Where to get this .conf dynamically?