Skip to content

PVC get FailedMount error for LocalPV hostpath PV #2767

New issue
New issue
Closed
Closed
PVC get FailedMount error for LocalPV hostpath PV#2767
Labels
platform/rancherIndicates an issue or PR related to the Rancher platformsubproject/hostpathIndicates an issue or PR related to the subproject: hostpath
@NexZhu

Description

@NexZhu
NexZhu
opened on Oct 6, 2019 · edited by NexZhu

Description

Failed to use LocalPV hostpath, please help!
Pod stuck in ContainerCreating and I got the following error in the log:

Events:
  Type     Reason       Age                 From                           Message
  ----     ------       ----                ----                           -------
  Normal   Scheduled    12m                 default-scheduler              Successfully assigned consul/consul-0 to internal-k8s-master1
  Warning  FailedMount  94s (x13 over 11m)  kubelet, MountVolume.NewMounter initialization failed for volume "pvc-4121ec0e-e815-11e9-a58e-00163e1274b8" : path "/data/openebs/local/pvc-4121ec0e-e815-11e9-a58e-00163e1274b8" does not exist
  Warning  FailedMount  93s (x5 over 10m)   kubelet, Unable to mount volumes for pod "consul-0_consul(49a26f4a-e81c-11e9-a58e-00163e1274b8)": timeout expired waiting for volumes to attach or mount for pod "consul"/"consul-0". list of unmounted volumes=[data]. list of unattached volumes=[data default-token-t5gfh]

But if I check the path on the node, the directory actually got created:

[root@ local]# pwd
/data/openebs/local
[root@ local]# ll
total 4
drwxrwxrwx 2 root root 4096 Oct  6 16:42 pvc-4121ec0e-e815-11e9-a58e-00163e1274b8

StorageClass "openebs-hostpath":

{
    "apiVersion": "storage.k8s.io/v1",
    "kind": "StorageClass",
    "metadata": {
        "annotations": {
            "cas.openebs.io/config": "- name: StorageType\n  value: \"hostpath\"\n- name: BasePath\n  value: \"/data/openebs/local\"\n",
            "openebs.io/cas-type": "local"
        },
        "creationTimestamp": "2019-10-05T04:18:51Z",
        "name": "openebs-hostpath",
        "resourceVersion": "290797",
        "selfLink": "/apis/storage.k8s.io/v1/storageclasses/openebs-hostpath",
        "uid": "3cb4438c-e727-11e9-8934-00163e1274b8"
    },
    "provisioner": "openebs.io/local",
    "reclaimPolicy": "Delete",
    "volumeBindingMode": "WaitForFirstConsumer"
}

PVC:

{
    "apiVersion": "v1",
    "kind": "PersistentVolumeClaim",
    "metadata": {
        "annotations": {
            "pv.kubernetes.io/bind-completed": "yes",
            "pv.kubernetes.io/bound-by-controller": "yes",
            "volume.beta.kubernetes.io/storage-provisioner": "openebs.io/local",
            "volume.kubernetes.io/selected-node": "internal-k8s-master1"
        },
        "creationTimestamp": "2019-10-06T08:42:39Z",
        "finalizers": [
            "kubernetes.io/pvc-protection"
        ],
        "labels": {
            "app": "consul",
            "release": "consul"
        },
        "name": "data-consul-0",
        "namespace": "consul",
        "resourceVersion": "292898",
        "selfLink": "/api/v1/namespaces/consul/persistentvolumeclaims/data-consul-0",
        "uid": "4121ec0e-e815-11e9-a58e-00163e1274b8"
    },
    "spec": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "resources": {
            "requests": {
                "storage": "8Gi"
            }
        },
        "storageClassName": "openebs-hostpath",
        "volumeMode": "Filesystem",
        "volumeName": "pvc-4121ec0e-e815-11e9-a58e-00163e1274b8"
    },
    "status": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "capacity": {
            "storage": "8Gi"
        },
        "phase": "Bound"
    }
}

PV:

{
    "apiVersion": "v1",
    "kind": "PersistentVolume",
    "metadata": {
        "annotations": {
            "pv.kubernetes.io/provisioned-by": "openebs.io/local"
        },
        "creationTimestamp": "2019-10-06T08:42:51Z",
        "finalizers": [
            "kubernetes.io/pv-protection"
        ],
        "labels": {
            "openebs.io/cas-type": "local-hostpath"
        },
        "name": "pvc-4121ec0e-e815-11e9-a58e-00163e1274b8",
        "resourceVersion": "292897",
        "selfLink": "/api/v1/persistentvolumes/pvc-4121ec0e-e815-11e9-a58e-00163e1274b8",
        "uid": "48602c57-e815-11e9-a58e-00163e1274b8"
    },
    "spec": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "capacity": {
            "storage": "8Gi"
        },
        "claimRef": {
            "apiVersion": "v1",
            "kind": "PersistentVolumeClaim",
            "name": "data-consul-0",
            "namespace": "consul",
            "resourceVersion": "292839",
            "uid": "4121ec0e-e815-11e9-a58e-00163e1274b8"
        },
        "local": {
            "fsType": "",
            "path": "/data/openebs/local/pvc-4121ec0e-e815-11e9-a58e-00163e1274b8"
        },
        "nodeAffinity": {
            "required": {
                "nodeSelectorTerms": [
                    {
                        "matchExpressions": [
                            {
                                "key": "kubernetes.io/hostname",
                                "operator": "In",
                                "values": [
                                    "internal-k8s-master1"
                                ]
                            }
                        ]
                    }
                ]
            }
        },
        "persistentVolumeReclaimPolicy": "Delete",
        "storageClassName": "openebs-hostpath",
        "volumeMode": "Filesystem"
    },
    "status": {
        "phase": "Bound"
    }
}

Expected Behavior

Mount successfully

Current Behavior

Got FailedMount error

Your Environment

  • kubectl get nodes:
NAME                   STATUS   ROLES                      AGE   VERSION
internal-k8s-master1   Ready    controlplane,etcd,worker   31h   v1.14.6
internal-k8s-worker1   Ready    etcd,worker                30h   v1.14.6
  • kubectl get pods --all-namespaces:
NAMESPACE       NAME                                                         READY   STATUS              RESTARTS   AGE
cattle-system   cattle-cluster-agent-7f4c9bb7b8-6p68t                        1/1     Running             5          31h
cattle-system   cattle-node-agent-s8swz                                      1/1     Running             1          30h
cattle-system   cattle-node-agent-x2jbh                                      1/1     Running             1          31h
cattle-system   kube-api-auth-ccg2b                                          1/1     Running             1          31h
consul          consul-0                                                     0/1     ContainerCreating   0          17m
kube-system     calico-node-6grt7                                            1/1     Running             1          30h
kube-system     calico-node-gfrrp                                            1/1     Running             1          31h
kube-system     coredns-autoscaler-5d5d49b8ff-jgwcn                          1/1     Running             1          31h
kube-system     coredns-bdffbc666-726zs                                      1/1     Running             3          31h
kube-system     metrics-server-7f6bd4c888-mpf7b                              1/1     Running             1          31h
kube-system     rke-coredns-addon-deploy-job-q82bl                           0/1     Completed           0          31h
kube-system     rke-metrics-addon-deploy-job-98hvv                           0/1     Completed           0          31h
kube-system     rke-network-plugin-deploy-job-thcp8                          0/1     Completed           0          31h
kube-system     tiller-deploy-79b98967c5-wrkdw                               1/1     Running             1          7h24m
kubeapps        apprepo-sync-bitnami-1570353600-nbvq6                        0/1     Completed           0          30m
kubeapps        apprepo-sync-bitnami-1570354200-9jzd9                        0/1     Completed           0          20m
kubeapps        apprepo-sync-bitnami-1570354800-twxjt                        0/1     Completed           0          10m
kubeapps        apprepo-sync-bitnami-1570355400-jggsq                        0/1     Completed           0          15s
kubeapps        apprepo-sync-bitnami-hsprv-l7tpb                             0/1     Completed           0          3h57m
kubeapps        apprepo-sync-bitnami-lnq8s-6mp9r                             0/1     Completed           0          26h
kubeapps        apprepo-sync-incubator-1570354200-pm5h8                      0/1     Completed           0          20m
kubeapps        apprepo-sync-incubator-1570354800-p4wc9                      0/1     Completed           0          10m
kubeapps        apprepo-sync-incubator-1570355400-wdgng                      0/1     Completed           0          15s
kubeapps        apprepo-sync-incubator-k472m-cxldf                           0/1     Completed           0          26h
kubeapps        apprepo-sync-incubator-lvfd8-vtwpz                           0/1     Completed           0          3h57m
kubeapps        apprepo-sync-stable-1570353600-fjrzh                         0/1     Completed           0          30m
kubeapps        apprepo-sync-stable-1570354200-7rkn8                         0/1     Completed           0          20m
kubeapps        apprepo-sync-stable-1570354800-qch8j                         0/1     Completed           0          10m
kubeapps        apprepo-sync-stable-1570355400-lxrgc                         1/1     Running             0          15s
kubeapps        apprepo-sync-stable-2prw6-tl92t                              0/1     Completed           0          3h57m
kubeapps        apprepo-sync-stable-v4kr4-kz294                              0/1     Completed           0          26h
kubeapps        apprepo-sync-svc-cat-1570354200-9ccs4                        0/1     Completed           0          20m
kubeapps        apprepo-sync-svc-cat-1570354800-fkd7d                        0/1     Completed           0          10m
kubeapps        apprepo-sync-svc-cat-1570355400-sbhbk                        0/1     Completed           0          15s
kubeapps        apprepo-sync-svc-cat-hxszc-r5wvm                             0/1     Completed           0          3h57m
kubeapps        kubeapps-6bd9495c56-8bxdt                                    1/1     Running             3          26h
kubeapps        kubeapps-6bd9495c56-mcx6g                                    1/1     Running             2          26h
kubeapps        kubeapps-internal-apprepository-controller-55fcd5966-72ln8   1/1     Running             1          26h
kubeapps        kubeapps-internal-chartsvc-7fc7bc4fc5-klzh4                  1/1     Running             10         26h
kubeapps        kubeapps-internal-chartsvc-7fc7bc4fc5-n729k                  1/1     Running             2          26h
kubeapps        kubeapps-internal-dashboard-5875d9f794-kgt2l                 1/1     Running             1          26h
kubeapps        kubeapps-internal-dashboard-5875d9f794-zqr9h                 1/1     Running             1          26h
kubeapps        kubeapps-internal-tiller-proxy-696ffcd799-76654              1/1     Running             3          26h
kubeapps        kubeapps-internal-tiller-proxy-696ffcd799-fkqqv              1/1     Running             3          26h
kubeapps        kubeapps-mongodb-6cbcc9ffd4-lg6m5                            1/1     Running             1          26h
openebs         openebs-admission-server-797586bf87-sgcpc                    1/1     Running             1          29h
openebs         openebs-apiserver-74bb8f8649-f22rs                           1/1     Running             1          29h
openebs         openebs-localpv-provisioner-7f9c88f575-rs865                 1/1     Running             22         29h
openebs         openebs-ndm-4d9l2                                            1/1     Running             1          29h
openebs         openebs-ndm-operator-55cc69dc96-9p6s9                        1/1     Running             2          29h
openebs         openebs-ndm-skq2m                                            1/1     Running             1          29h
openebs         openebs-provisioner-78b964ffb9-2wskm                         1/1     Running             1          29h
openebs         openebs-snapshot-operator-56c69dcb58-7z2vb                   2/2     Running             3          29h
  • kubectl get sc:
NAME                                PROVISIONER                                                AGE
openebs-device                      openebs.io/local                                           29h
openebs-hostpath                    openebs.io/local                                           29h
openebs-jiva-default                openebs.io/provisioner-iscsi                               29h
openebs-jiva-disk-2repl (default)   openebs.io/provisioner-iscsi                               3h5m
openebs-snapshot-promoter           volumesnapshot.external-storage.k8s.io/snapshot-promoter   29h
  • kubectl get pv:
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS       REASON   AGE
pvc-4121ec0e-e815-11e9-a58e-00163e1274b8   8Gi        RWO            Delete           Bound    consul/data-consul-0   openebs-hostpath            68m
  • kubectl get pvc:
NAMESPACE   NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
consul      data-consul-0   Bound    pvc-4121ec0e-e815-11e9-a58e-00163e1274b8   8Gi        RWO            openebs-hostpath   69m
  • OS (from /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (from uname -a):
Linux internal-k8s-master1 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
    Rancher

Activity

AmitKumarDas

AmitKumarDas commented on Oct 6, 2019

@AmitKumarDas
AmitKumarDas
on Oct 6, 2019

Cc @kmova

ranjithwingrider

ranjithwingrider commented on Oct 6, 2019

@ranjithwingrider
ranjithwingrider
on Oct 6, 2019
Contributor

Can you check the prerequistes mentioned in the documentation?
https://docs.openebs.io/docs/next/prerequisites.html#rancher

NexZhu

NexZhu commented on Oct 6, 2019

@NexZhu
NexZhu
on Oct 6, 2019
Author

Can you check the prerequistes mentioned in the documentation?
https://docs.openebs.io/docs/next/prerequisites.html#rancher

Step 2: If you are using Jiva or Local PV for provisioning OpenEBS volume, Add extra_binds under kubelet service in cluster YAML. If the volume is using a mounted path on the host, then you must add the mounted path under extra_binds section.

services:
  kubelet:
    extra_binds:
     - /var/openebs/local:/var/openebs/local

This fixed it for me, thanks!

NexZhu
closed this as completedon Oct 6, 2019
AmitKumarDas
added
extra-binds 💪
subproject/hostpathIndicates an issue or PR related to the subproject: hostpath
platform/rancherIndicates an issue or PR related to the Rancher platform
on Oct 7, 2019
BegoneRaven

BegoneRaven commented on Sep 9, 2020

@BegoneRaven
BegoneRaven
on Sep 9, 2020 · edited by BegoneRaven

Just in case if anyone is wondering what path is to given in extra_binds, if you have a non default or custom host path like
/data/openebs/local for storage
This is what worked for me.

services:
  kubelet:
    extra_binds:
     -/data/openebs/local:/data/openebs/local

Both paths are supposed to be the same!
If you are using helm for openebs installation, ensure that .basepath is passed as an additional parameter
helm install --debug -n openebs openebs openebs/openebs --set localprovisioner.basePath="/data/openebs/local"

maxisam

maxisam commented on Feb 8, 2022

@maxisam
maxisam
on Feb 8, 2022

In my case, reboot fix it.

Rammurthy5

Rammurthy5 commented on Apr 27, 2024

@Rammurthy5
Rammurthy5
on Apr 27, 2024

I have the same problem but on Talos Linux. Can i ask how to fix this, please?

letajmal

letajmal commented on Jun 19, 2024

@letajmal
letajmal
on Jun 19, 2024 · edited by letajmal

I have the same problem but on Talos Linux. Can i ask how to fix this, please?

For talos, modify worker.yaml
under kubelet

    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.30.1 # The `image` field is an optional reference to an alternative kubelet image.
        defaultRuntimeSeccompProfileEnabled: true # Enable container runtime default Seccomp profile.
        disableManifestsDirectory: true # The `disableManifestsDirectory` field configures the kubelet to get static pod manifests from the /etc/kubernetes/manifests directory.
        
        # # The `ClusterDNS` field is an optional reference to an alternative kubelet clusterDNS ip list.
        # clusterDNS:
        #     - 10.96.0.10
        #     - 169.254.2.53

        # # The `extraArgs` field is used to provide additional flags to the kubelet.
        # extraArgs:
        #     key: value

        # # The `extraMounts` field is used to add additional mounts to the kubelet container.
        extraMounts:

        # for etcd
            - destination: /var/local/mayastor/localpv-hostpath/etcd # Destination is the absolute path where the mount will be placed in the container.
              type: bind # Type specifies the mount kind.
              source: /var/local/mayastor/localpv-hostpath/etcd # Source specifies the source path of the mount.
        #       # Options are fstab style mount options.
              options:
                - bind
                - rshared
                - rw

        # for loki
            - destination: /var/local/mayastor/localpv-hostpath/loki # Destination is the absolute path where the mount will be placed in the container.
              type: bind # Type specifies the mount kind.
              source: /var/local/mayastor/localpv-hostpath/loki # Source specifies the source path of the mount.
        #       # Options are fstab style mount options.
              options:
                - bind
                - rshared
                - rw

then, apply these changes to all the worker nodes

talosctl -n <worker node IP> apply-config --file worker.yaml

ref: https://www.talos.dev/v1.7/kubernetes-guides/configuration/local-storage/

Mayastor Documentation in Talos site is now deprecated, it is only valid for mayastor version < 2.5. For later versions, refer: https://openebs.io/docs/quickstart-guide/installation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    platform/rancherIndicates an issue or PR related to the Rancher platformsubproject/hostpathIndicates an issue or PR related to the subproject: hostpath

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @maxisam@AmitKumarDas@NexZhu@BegoneRaven@ranjithwingrider

        Issue actions
          PVC get FailedMount error for LocalPV hostpath PV · Issue #2767 · openebs/openebs