Skip to content

Error creating rbd image: executable file not found in $PATH #38923

Closed
@sbezverk

Description

@sbezverk
Contributor

Kubernetes 1.5.1 All in One on the baremetal, external ceph cluster.

Using StorageClass and PVC to provision volume.
PVC is stuck in Pending state and log indicates the following error.

Dec 17 09:43:08 kube-1 journal: E1217 14:43:08.506433 1 rbd_util.go:341] rbd: Error creating rbd image: executable file not found in $PATH
Dec 17 09:43:08 kube-1 journal: E1217 14:43:08.506441 1 rbd.go:310] rbd: create volume failed, err: executable file not found in $PATH

RBD executable IS in PATH as you can see from the output below.
[root@kube-1 local-cluster]# which rbd
/usr/bin/rbd

If it is looking for some other executable then question 1: which one exactly? and 2: the error message should specify the name of executable which is missing.

Activity

thanodnl

thanodnl commented on Dec 18, 2016

@thanodnl

I ran into the same issue and found it strange that kubernetes complained about rbd not being on my $PATH. After digging around I have found why it is complaining about missing rbd.

The process that I have tracked the log line down to is the kube-controller-manager. This process is running in its own pod (atleast after installing a cluster with kubeadm). The container in this pod is missing the rbd binary.

I have tried creating a new image from the released kube-controller-manager image and add rbd to it. It was no surprise to find that it didn't work as rbd probably needs some other shared libraries that I didn't copy over. At least rbd was now in my container but non-runnable and that got picked up by kube-controller-manager.

Is there a way to add an other container to the kube-controller-manager pod, a container that has a working rbd binary in it? Like the container used in https://github.com/ceph/ceph-docker/tree/master/examples/kubernetes

sbezverk

sbezverk commented on Dec 18, 2016

@sbezverk
ContributorAuthor

Yep, you are right. rbd binary does not exist in kube-controller-manager container.
docker exec -tu root -ti 1dd53ef015ab find / -name rbd
/sys/bus/rbd
/sys/devices/rbd
/sys/module/rbd
/sys/module/libceph/holders/rbd

v1k0d3n

v1k0d3n commented on Dec 18, 2016

@v1k0d3n

@thanodnl we had to create a container with kube-controller-manager binary packed in for this purpose. this is a workaround, but you are free to use our image until a true "fix" is in place.

https://quay.io/repository/attcomdev/kube-controller-manager?tab=tags

we've been editing the "image" in /etc/kubernetes/manifests/kube-controller-manager.json post deployment, and that seems to do the trick for us.

let us know if this works for you too?

thanodnl

thanodnl commented on Dec 18, 2016

@thanodnl

I have just started your image and got it to work! Thanks a lot.

Just one question, your image wasn't able to do a lookup of the monitor on its dns name via skyDNS. For now I have put the monitor ip's in the storage class and it works.

Just wondering if you have tried using this image in combination with the ceph monitor running on k8s.

v1k0d3n

v1k0d3n commented on Dec 18, 2016

@v1k0d3n

@thanodnl glad it worked (with some tinkering). we have it working just fine as described in our project at: https://github.com/att-comdev/openstack-helm.

if there is anything you find missing, please submit an issue and we can address it. we're at the very beginning of the project, but will be improving it quite a bit over the coming months. ceph is first class in our solution, so these issues need addressed. if you catch something, either submit an issue, or we'd be happy to accept a PR as well. the docker container is listed here: https://github.com/att-comdev/dockerfiles/tree/master/kube-controller-manager. i have plans to look over submitting these 'fixes' in kubernetes proper; i just started looking into it this afternoon.

v1k0d3n

v1k0d3n commented on Dec 18, 2016

@v1k0d3n

adding @alanmeadows to the thread (keep an eye on this one).

kongslund

kongslund commented on Jan 19, 2017

@kongslund

Thanks, @v1k0d3n. Any progress on a more long term solution?

v1k0d3n

v1k0d3n commented on Jan 20, 2017

@v1k0d3n

@kongslund i think the long term goal is to build a custom kube-controller with the pvc tooling you need. i don't want to take over @sbezverk's issue, but I think the reason the executable isn't found because it hasn't been included. we created a kube-controller image with the pvc bins required for ceph. you're welcome to use that, or you may want to create one (based on our example above) for your own needs.

alanmeadows

alanmeadows commented on Jan 20, 2017

@alanmeadows

@thanodnl What happens is that the k8s controller manager leverages hostNetwork: true so it essentially inherits the underlying physical hosts /etc/resolv.conf. We resolve this today by ensuring all k8s nodes leverage skyDNS as their first resolution server prior to starting kubelet with fallbacks to standard resolution endpoints.

The correct solution here for which I'm sure there is an open issue is to support dnsPolicy: ClusterFirst on hostNetwork: true hosts. To my knowledge, this has no effect (nor does it error at the API level) when leveraged on hostNetwork enabled artifacts.

At the end of the day, ensuring skyDNS is leveraged from the controller pod allows us to resolve ceph-mon.ceph (a k8s deployment of ceph itself) from the controller manager so we can avoid leveraging the monitor IPs directly.

4admin2root

4admin2root commented on Apr 27, 2017

@4admin2root

Is there any plan to add ceph rbd to google_containers docker image (e.g, gcr.io/google_containers/kube-controller-manager-amd64) ?

jingxu97

jingxu97 commented on May 3, 2017

@jingxu97
Contributor
luxas

luxas commented on May 30, 2017

@luxas
Member

cc @kubernetes/sig-storage-bugs

I guess and think this is out of scope to include rbd, but very unfortunate bug indeed.

jsafrane

jsafrane commented on May 30, 2017

@jsafrane
Member

I am working on kubernetes/enhancements#278 - /usr/bin/rbd (and similar tools) could run in containers. On GKE you would run a daemonset with all Ceph utilities and you won't need anything on the nodes nor master(s).

52 remaining items

dims

dims commented on May 27, 2018

@dims
Member

/close

@sbezverk please reopen if necessary

feresberbeche

feresberbeche commented on May 28, 2018

@feresberbeche

/reopen
/usr/bin # yum list |grep ceph-common root@kubernetes ceph-common.x86_64 1:10.2.10-0.el7 @Ceph
/usr/bin # ceph -v root@kubernetes ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe)

kubectl version root@kubernetes Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-18T23:58:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
I0528 13:39:01.708902 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ceph-pvc", UID:"48891a51-6272-11e8-8ff1-e83935e84dc8", APIVersion:"v1", ResourceVersion:"959049", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "rbd": failed to create rbd image: executable file not found in $PATH, command output:

I still have this problem, I tried to use an external provisioner and I had different problem
I hhave the rdb binary and kubelet binary inside the node @ /usr/bin/

k8s-ci-robot

k8s-ci-robot commented on May 28, 2018

@k8s-ci-robot
Contributor

@feresberbeche: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

shuraa

shuraa commented on Dec 22, 2018

@shuraa

@cofyc

hi, guys,

Now you can avoid using customized kube-controller image, external-storage out-of-tree RBD provisioner is merged, you can use it instead. Here is guide:

1, Deploy standalone rbd-provisioner controller:

Ok, It can work. But what about of several already provisioned PVC/PV by kubernetes.io/rbd provisioner? How can I switch these volumes under the control of external storage controller? What is a clear upgrade path from 1.11 to 1.12?

cofyc

cofyc commented on Dec 22, 2018

@cofyc
Member

But what about of several already provisioned PVC/PV by kubernetes.io/rbd provisioner? How can I switch these volumes under the control of external storage controller?

There is no automatic way, but you can do it manually.

  • Create new claim and pre-bind old volume with it (set spec.Volume.Name on creation, otherwise provisioner will provisioner a new volume for it)
  • Update old volume spec.claimRef filed to point to new claim
  • Delete old claim

What is a clear upgrade path from 1.11 to 1.12?

No extra action required for RBD provisioning.

r0bj

r0bj commented on Jan 7, 2019

@r0bj

External provisioner is great during provisioning volumes but if you want to resize those volumes it's handled by in-tree volume plugin (kube-controller-manager):

Error expanding volume "volume-name" of plugin kubernetes.io/rbd : rbd info failed, error: executable file not found in $PATH

So if you want to resize volumes you still need to have rbd binary in kube-controller-manager image.

exodussystem

exodussystem commented on Feb 28, 2019

@exodussystem

@gyliu513

hi, there are two parts:

* Volume Provisioning: Currently, if you want dynamic provisioning, RBD provisioner in `controller-manager` needs to access `rbd` binary to create new image in ceph cluster for your PVC.
  [external-storage](https://github.com/kubernetes-incubator/external-storage) plans to move volume provisioners from in-tree to out-of-tree, there will be a separated RBD provisioner container image with `rbd` utility included ([kubernetes-incubator/external-storage#200](https://github.com/kubernetes-incubator/external-storage/issues/200)), then `controller-manager` do not need access `rbd` binary anymore.

* Volume Attach/Detach: `kubelet` needs to access `rbd` binary to attach (`rbd map`) and detach (`rbd unmap`) RBD image on node. If `kubelet` is running on the host, host needs to install `rbd` utility (install `ceph-common` package on most Linux distributions).

This is very useful detail explanations.

I installed Kubernetes 1.13.2 with Kubespray

For the first part (Volume Provisioning), kubespray deployment uses /external-storage/rbd-provisioner container. So it works fine. I checked the logs and Ceph created the volume ok

In the second part (Volume Attach/Detach), it failed because rbd binary was not found.

My kubelet is running on CoreOS host (not inside docker/rkt) because and due to to security, kubespray kubelet_deployment_type default is 'host'. I could NOT install ceph-common when setting up these nodes. However, I could invoke 'rbd' binary mounted from ceph/ceph container

Source:
https://stackoverflow.com/questions/37996983/invoking-rbd-docker-from-kubernetes-on-coreos-returns-fork-exec-invalid-argume

I still got the error stating that rbd could not find ceph.conf in these nodes. Where to get this .conf dynamically?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

sig/storageCategorizes an issue or PR as relevant to SIG Storage.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @farcaller@dims@cofyc@thanodnl@michaelfig

      Issue actions

        Error creating rbd image: executable file not found in $PATH · Issue #38923 · kubernetes/kubernetes