Skip to content

Kubeadm unknown service runtime.v1alpha2.RuntimeService #4581

New issue
New issue
Closed
Closed
Kubeadm unknown service runtime.v1alpha2.RuntimeService#4581
Labels
kind/question
@niklastojts

Description

@niklastojts
niklastojts
opened on Sep 24, 2020 · edited by niklastojts

Problem
Following Kubernetes official installation instruction for containerd and kubeadm init will fail with unknown service runtime.v1alpha2.RuntimeService.

# Commands from https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd
apt-get update && apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
apt-get update && apt-get install -y containerd.io

# Configure containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
systemctl restart containerd

kubeadm init
...
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR CRI]: container runtime is not running: output: time="2020-09-24T11:49:16Z" level=fatal msg="getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1

Solution:

rm /etc/containerd/config.toml
systemctl restart containerd
kubeadm init

Versions:

  • Ubuntu 20.04 (focal)
  • containerd.io 1.3.7
  • kubectl 1.19.2
  • kubeadm 1.19.2
  • kubelet 1.19.2

Activity

niklastojts
added
kind/bug
on Sep 24, 2020
fuweid

fuweid commented on Sep 25, 2020

@fuweid
fuweid
on Sep 25, 2020
Member

@blinkiz could you please provide your containerd config.toml and command result of ctr plugin ls? thanks

fuweid
added
kind/question
and removed
kind/bug
on Sep 25, 2020
niklastojts

niklastojts commented on Sep 25, 2020

@niklastojts
niklastojts
on Sep 25, 2020 · edited by niklastojts
Author

Apparently it is my fault this time. My ansible playbook did not override config.toml file as I expected. Am sorry for taking up your time, default installation instructions work great.

In the config.toml file installed by package containerd.io there is the line disabled_plugins = ["cri"] that am guessing creating the issue. That maybe is bad default setting to have in the package containerd.io but that is for another issue/bug.

Closing.

niklastojts
closed this as completedon Sep 25, 2020
mikebrow

mikebrow commented on Sep 25, 2020

@mikebrow
mikebrow
on Sep 25, 2020
Member

Apparently it is my fault this time. My ansible playbook did not override config.toml file as I expected. Am sorry for taking up your time, default installation instructions work great.

In the config.toml file installed by package containerd.io there is the line disabled_plugins = ["cri"] that am guessing creating the issue. That maybe is bad default setting to have in the package containerd.io but that is for another issue/bug.

Closing.

Docker (by default) uses that config for the containerd they install via containerd.io packages. Grr. been causing similar ^ issues for k8s users for years. :-)

alexcpn

alexcpn commented on Nov 25, 2020

@alexcpn
alexcpn
on Nov 25, 2020 · edited by alexcpn

I followed the official instructions here https://kubernetes.io/docs/setup/production-environment/container-runtimes/

and I was getting similar error

root@green-1:~# kubeadm init  --config=config.yaml 
W1125 12:58:32.733485   26426 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.4
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2020-11-25T12:58:32Z" level=fatal msg="getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

I checked /etc/containerd/config.toml and saw 'disabled_plugins = []'

Note the only thing I changed in the config.toml was to use systemd as true -it was different from the way docs has mentioned
(maybe this was the problem?)

[plugins."io.containerd.grpc.v1.cri"]
    systemd_cgroup = false --> to true

from docs

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

deleting this config.toml as given in the first post and restarting containerd service solved and kubeadm could proceed

alexcpn
mentioned this on Nov 25, 2020
alexcpn

alexcpn commented on Nov 25, 2020

@alexcpn
alexcpn
on Nov 25, 2020 · edited by alexcpn

This is definitely a bug. For the worker node I followed the docs exact and I was getting this error

Nov 25 14:01:37 green-3 kubelet[24158]: E1125 14:01:37.781735   24158 kubelet.go:2103] Container runtime network not ready: NetworkReady=false reason:Ne
Nov 25 14:01:40 green-3 kubelet[24158]: W1125 14:01:40.121655   24158 manager.go:1168] Failed to process watch event {EventType:0 Name:/kubepods/burstab
Nov 25 14:01:40 green-3 kubelet[24158]: I1125 14:01:40.161343   24158 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 858e1a9
Nov 25 14:01:41 green-3 kubelet[24158]: I1125 14:01:41.162982   24158 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 858e1a9
Nov 25 14:01:41 green-3 kubelet[24158]: I1125 14:01:41.163235   24158 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a46854a
Nov 25 14:01:41 green-3 kubelet[24158]: E1125 14:01:41.163527   24158 pod_workers.go:191] Error syncing pod 4dabce76-ceb5-43fb-bef1-1992a3aa124d ("kube-
Nov 25 14:01:41 green-3 kubelet[24158]: W1125 14:01:41.624869   24158 manager.go:1168] Failed to process watch event {EventType:0 Name:/kubepods/burstab
Nov 25 14:01:42 green-3 kubelet[24158]: I1125 14:01:42.164906   24158 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a46854a
Nov 25 14:01:42 green-3 kubelet[24158]: E1125 14:01:42.165198   24158 pod_workers.go:191] Error syncing pod 4dabce76-ceb5-43fb-bef1-1992a3aa124d ("kube-
Nov 25 14:01:43 green-3 kubelet[24158]: W1125 14:01:43.127317   24158 manager.go:1168] Failed to process watch event {EventType:0 Name:/kubepods/burstab
Nov 25 14:01:55 green-3 kubelet[24158]: I1125 14:01:55.011131   24158 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a46854a
Nov 25 14:02:17 green-3 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Nov 25 14:02:17 green-3 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Nov 25 14:02:17 green-3 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Nov 25 14:02:17 green-3 kubelet[25982]: I1125 14:02:17.474559   25982 server.go:411] Version: v1.19.4

till I deleted the config,toml restarted contained and kubelet, after which only the worker joined

fuweid

fuweid commented on Nov 26, 2020

@fuweid
fuweid
on Nov 26, 2020
Member

@alexcpn please open new issue to describe the case, thanks.

oldthreefeng
mentioned this on Jan 12, 2021
oldthreefeng
mentioned this on Jan 20, 2021
floryut
mentioned this on Apr 12, 2021
dmitry-irtegov
mentioned this on Apr 15, 2021

41 remaining items

Jparmo

Jparmo commented on Feb 24, 2023

@Jparmo
Jparmo
on Feb 24, 2023 · edited by Jparmo

I have the same problem when I try to join to master node and my containerd is activate, nothing works
version of kubelet=1.25.5-00
[preflight] Running pre-flight checks
[WARNING SystemVerification]: missing optional cgroups: blkio
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR CRI]: container runtime is not running: output: time="2023-02-24T20:01:08Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint "unix:///var/run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

hamedsol

hamedsol commented on Feb 25, 2023

@hamedsol
hamedsol
on Feb 25, 2023

Follow official documents regarding containerd to enable SystemCgroups on etc/containerd/config.toml if you don't have this file make it by sudo containerd config default | tee etc/containerd/configuration.toml , better to install containerd from binary rather than by package manager

Enoxime

Enoxime commented on Feb 26, 2023

@Enoxime
Enoxime
on Feb 26, 2023

Got the same issue and fixed it by installing the containerd.io package from the docker repository instead of the one from ubuntu's repository.
see: https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository

I have ubuntu 22.04.2 on VMs and raspberry pies

Also it seems there is presently an issue to retrieve the gpg key from https://packages.cloud.google.com/apt/doc/apt-key.gpg

zsinba

zsinba commented on Mar 1, 2023

@zsinba
zsinba
on Mar 1, 2023

Got the same issue. there is no answer in debian 11.

riyazhakki

riyazhakki commented on Mar 14, 2023

@riyazhakki
riyazhakki
on Mar 14, 2023

I was having same issue on ubutnu 20.04.3, it got fixed after reinstalling docker,containrd, runc etc, then deleting the /etc/container.d/config.toml file.

thallgren

thallgren commented on Apr 7, 2023

@thallgren
thallgren
on Apr 7, 2023

I struggled with this too. The solution for me was to comment this line out in /etc/containerd/config.toml

disabled_plugins = ["cri"]
nivaran

nivaran commented on Apr 8, 2023

@nivaran
nivaran
on Apr 8, 2023 · edited by nivaran

Refer to Three pages of Kubernetes Doc:

  1. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
  2. https://kubernetes.io/docs/setup/production-environment/container-runtimes/
  3. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Steps :

  1. complete the Pre-requisite ( pages 1 & 2 )
  2. install CRI from page 2 ( For Containered & Docker- https://docs.docker.com/engine/install/ubuntu/ )
  3. Make changes in /etc/containerd/config.toml ( page 2 ) then restart containerd
  4. Install kubectl, kubeadm, kubelet ( page 1 ) then systemctl daemon-reload
  5. kubeadm init ( page 3 )
  6. install CNI ( for weave net - https://www.weave.works/docs/net/latest/kubernetes/kube-addon/)
  7. systemctl daemon-reload && kubectl get nodes

OR can use single shell script for both master-node & worker node configuration

script Link - https://github.com/nivaran/Install_k8s_using_shell_script

shaharby7
added a commit that references this issue on Apr 14, 2023

fix(debian kube-mast): 🐛 fix unknown service runtime.v1alpha2.Runtime…

66f0877
shaharby7
mentioned this on Apr 14, 2023
hub4ops

hub4ops commented on May 12, 2023

@hub4ops
hub4ops
on May 12, 2023

Hi folks,
my bash history from the new control plane is below, it works.

   18  apt update
   19  apt upgrade -y
   20  reboot
   21  cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

   22  sudo modprobe overlay
   23  sudo modprobe br_netfilter
   24  cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

   25  sudo sysctl --system
   26  sudo apt-get update
   27  wget https://github.com/containerd/containerd/releases/download/v1.7.1/containerd-1.7.1-linux-amd64.tar.gz
   28  tar Cxzvf /usr/local containerd-1.7.1-linux-amd64.tar.gz
   29  systemctl daemon-reload
   30  systemctl enable --now containerd
   31  nano /usr/local/lib/systemd/system/containerd.service
   32  systemctl status sshd
   33  ls -lh /lib/systemd/system/
   34  nano /lib/systemd/system/containerd.service
   35  systemctl daemon-reload
   36  systemctl status containerd.service
   37  nano /lib/systemd/system/containerd.service
   38  ls /usr/local/bin/containerd
   39  systemctl enable --now containerd
   40  wget https://github.com/opencontainers/runc/releases/download/v1.1.7/runc.amd64
   41  install -m 755 runc.amd64 /usr/local/sbin/runc
   42  wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
   43  tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz
   44  systemctl status containerd.service
   45  systemctl restart containerd.service
   46  systemctl status containerd.service
   47  containerd -v
   48  sudo mkdir -p /etc/containerd
   49  sudo containerd config default | sudo tee /etc/containerd/config.tom
   50  sudo containerd config default | sudo tee /etc/containerd/config.toml
   51  ls /etc/containerd/
   52  rm /etc/containerd/config.tom
   53  sudo systemctl restart containerd
   54  sudo systemctl status containerd
   55  sudo swapoff -a
   56  nano /etc/fstab
   57  sudo apt-get update && sudo apt-get install -y apt-transport-https curl
   58  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
   59  cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

   60  sudo apt-get update
   61  sudo apt-get install -y kubelet=1.26.0-00 kubeadm=1.26.0-00 kubectl=1.26.0-00
   62  sudo apt-mark hold kubelet kubeadm kubectl
   63  sudo kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.26.0
arun-chib

arun-chib commented on Jun 25, 2023

@arun-chib
arun-chib
on Jun 25, 2023

It got fixed for me finally:

fix:

Update contained to the latest and fix toml file with below changes:

disabled_plugins = ["cri"] -->> disabled_plugins = [""]

Done.
Thnak me later ;)

CivenResun
mentioned this on Sep 6, 2023
alanlupsha

alanlupsha commented on Feb 7, 2024

@alanlupsha
alanlupsha
on Feb 7, 2024

February 2024 fix:

sudo containerd config default > /etc/containerd/config.toml

sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/' /etc/containerd/config.toml
sudo sed -i 's/snapshotter = "overlayfs"/snapshotter = "native"/' /etc/containerd/config.toml
sudo sed -i 's/systemd_cgroup \= true/systemd_cgroup \= true/' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl status containerd
q

kubectl get all
kubectl get nodes
bjrara

bjrara commented on Apr 23, 2024

@bjrara
bjrara
on Apr 23, 2024

February 2024 fix:

sudo containerd config default > /etc/containerd/config.toml

sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/' /etc/containerd/config.toml
sudo sed -i 's/snapshotter = "overlayfs"/snapshotter = "native"/' /etc/containerd/config.toml
sudo sed -i 's/systemd_cgroup \= true/systemd_cgroup \= true/' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl status containerd
q

kubectl get all
kubectl get nodes

This fix works for me! @alanlupsha Would you mind sharing how you came up with this fix?

alanlupsha

alanlupsha commented on Apr 23, 2024

@alanlupsha
alanlupsha
on Apr 23, 2024

This fix works for me! @alanlupsha Would you mind sharing how you came up with this fix?

I'm glad that works! I used them googles on the interwebs and put all the findings into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/question

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @thallgren@nodesocket@felipecrp@zsinba@alexcpn

        Issue actions
          Kubeadm unknown service runtime.v1alpha2.RuntimeService · Issue #4581 · containerd/containerd