kubelet: add initial support for cgroupv2 #85218

giuseppe · 2019-11-13T15:22:09Z

/kind feature

do a conversion from the cgroups v1 limits to cgroups v2.

e.g. cpu.shares on cgroups v1 has a range of [2-262144] while the
equivalent on cgroups v2 is cpu.weight that uses a range [1-10000].

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

Does this PR introduce a user-facing change?:

NONE

Support for running on a host that uses cgroups v2 unified mode

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2019-11-13T15:22:17Z

Hi @giuseppe. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/kubelet/cm/helpers_linux.go

mattjmcnaughton

/uncc

Looking forward to reading the KEP :) Removing myself as a reviewer while this diff is WIP.

mrunalp · 2019-11-14T17:45:19Z

/ok-to-test

giuseppe · 2019-11-18T14:54:21Z

Looking forward to reading the KEP :) Removing myself as a reviewer while this diff is WIP.

KEP here: kubernetes/enhancements#1370

WIP backport of: kubernetes/kubernetes#85218 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

derekwaynecarr · 2020-03-12T14:48:08Z

/assign

derekwaynecarr

i want to review this a little closer.

no conversion was needed for pids and hugepages, correct?

derekwaynecarr · 2020-03-12T14:51:33Z

pkg/kubelet/cm/cgroup_manager_linux.go

-			Cgroups: cgroups,
-			Paths:   paths,
-		}, nil
+		f, err := cgroupsystemd.NewSystemdCgroupsManager()


does this manager work on v1 and v2 hosts?

i need to check the runc code, but is this abstracing legacy from unified manager?

yes, NewSystemdCgroupsManager() checks for the cgroup version used:

https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go#L118-L136

If we are running on cgroup v1, it will use cgroupsystemd.LegacyManager

thanks for pointer, and noting that i confirmed this is in the vendored version.

giuseppe · 2020-03-12T15:01:48Z

no conversion was needed for pids and hugepages, correct?

yes correct, both pids and hugetlb don't need any conversion. I'll need to add support for hugetlb to cAdvisor though, once Linux 5.6 is out (it is still at 5.6-rc5), support in libcontainer is already being added.

mattjmcnaughton · 2020-03-14T16:04:11Z

Cool stuff!
It will take me a couple of days to give this a full review (gotta read up on cgroup v2 a bit first :) ).

@giuseppe I haven't forgotten about this :) I just haven't had a lot of time to review larger PRs recently :) Will hopefully have thoughts by end of week. Thanks for your patience.

/uncc

Whoops, I didn't get to this before @derekwaynecarr did. Tbh, I don't see myself adding anything that his review wouldn't, so going to defer to him on the review :)

derekwaynecarr · 2020-03-26T15:08:46Z

pkg/kubelet/cm/cgroup_manager_linux.go

+	if libcontainercgroups.IsCgroup2UnifiedMode() {
+		if resourceConfig.CpuShares != nil {
+			// Convert from the range [2-262144] to [1-10000]
+			resources.CpuWeight = (1 + ((*resourceConfig.CpuShares-2)*9999)/262142)


minor nit: may be useful to move this conversion into a helper that we can then unit test in a follow-on.

PR here: #89567

derekwaynecarr

great to see this progress.

/approve
/lgtm

derekwaynecarr · 2020-03-26T15:09:50Z

/milestone v1.19

k8s-ci-robot · 2020-03-26T15:14:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, giuseppe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/cm/OWNERS~~ [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mattjmcnaughton · 2020-03-26T23:39:20Z

Wow, exciting stuff @giuseppe @derekwaynecarr - big thank you for all your work!

follow-up for kubernetes#85218 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

k8s-ci-robot requested review from dashpole and mattjmcnaughton November 13, 2019 15:23

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 13, 2019

AkihiroSuda reviewed Nov 13, 2019

View reviewed changes

pkg/kubelet/cm/helpers_linux.go Show resolved Hide resolved

AkihiroSuda reviewed Nov 13, 2019

View reviewed changes

pkg/kubelet/cm/helpers_linux.go Outdated Show resolved Hide resolved

giuseppe force-pushed the cgroupv2 branch from 83eb7a5 to 1e1cbdf Compare November 13, 2019 15:44

mattjmcnaughton reviewed Nov 14, 2019

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 14, 2019

alban mentioned this pull request Dec 4, 2019

Gardenlet gardener/gardener#1601

Merged

giuseppe force-pushed the cgroupv2 branch from 1e1cbdf to 4969d96 Compare December 4, 2019 17:54

giuseppe mentioned this pull request Dec 9, 2019

add KEP for cgroups v2 support kubernetes/enhancements#1370

Merged

giuseppe force-pushed the cgroupv2 branch from 4969d96 to 0704ef6 Compare December 19, 2019 19:43

giuseppe force-pushed the cgroupv2 branch from 0704ef6 to 1413203 Compare January 28, 2020 10:44

giuseppe added a commit to giuseppe/origin that referenced this pull request Jan 28, 2020

UPSTREAM: <carry>: cgroupv2 support

5b598c3

WIP backport of: kubernetes/kubernetes#85218 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe mentioned this pull request Jan 28, 2020

[WIP] UPSTREAM: <carry>: cgroupv2 support openshift/origin#24467

Closed

giuseppe force-pushed the cgroupv2 branch from 1413203 to 4cd7cf1 Compare January 28, 2020 14:27

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 12, 2020

k8s-ci-robot assigned derekwaynecarr Mar 12, 2020

derekwaynecarr reviewed Mar 12, 2020

View reviewed changes

neolit123 mentioned this pull request Mar 14, 2020

Add guide to switch to cgroup v1 kubernetes/website#19632

Closed

derekwaynecarr reviewed Mar 26, 2020

View reviewed changes

derekwaynecarr approved these changes Mar 26, 2020

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 26, 2020

k8s-ci-robot added this to the v1.19 milestone Mar 26, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 26, 2020

k8s-ci-robot merged commit 34c8b26 into kubernetes:master Mar 26, 2020

giuseppe mentioned this pull request Mar 27, 2020

kubelet: add tests for cgroup v2 conversions #89567

Merged

giuseppe added a commit to giuseppe/kubernetes that referenced this pull request Mar 27, 2020

kubelet: add tests for cgroup v2 conversions

c4429d8

follow-up for kubernetes#85218 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

AkihiroSuda mentioned this pull request Mar 28, 2020

update components rootless-containers/usernetes#153

Merged

harche mentioned this pull request Mar 30, 2020

Add a crio tests kubernetes/test-infra#17014

Merged

neolit123 mentioned this pull request Apr 1, 2020

Support for cgroupv2 kubernetes/system-validators#12

Merged

neolit123 mentioned this pull request May 5, 2020

Kubelet on Fedora 31+ doesn't start without kernel parameter systemd.unified_cgroup_hierarchy = 0 #90710

Closed

giuseppe mentioned this pull request Jan 12, 2021

cgroup v2 kubernetes/enhancements#2254

Closed

4 tasks

BenTheElder mentioned this pull request Feb 23, 2021

Kubelet for versions prior to 1.19 do not start under unified cgroups kubernetes-sigs/kind#2082

Closed

DennisGlindhart mentioned this pull request Apr 26, 2021

Cluster join fails due to insufficient ephemeral-storage #99305

Closed

kakkoyun mentioned this pull request Sep 2, 2021

Add support for cgroup v2 parca-dev/parca-agent#23

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubelet: add initial support for cgroupv2 #85218

kubelet: add initial support for cgroupv2 #85218

giuseppe commented Nov 13, 2019 •

edited

k8s-ci-robot commented Nov 13, 2019

mattjmcnaughton left a comment

mrunalp commented Nov 14, 2019

giuseppe commented Nov 18, 2019

derekwaynecarr commented Mar 12, 2020

derekwaynecarr left a comment

derekwaynecarr Mar 12, 2020

derekwaynecarr Mar 12, 2020

giuseppe Mar 12, 2020

derekwaynecarr Mar 26, 2020

giuseppe commented Mar 12, 2020

mattjmcnaughton commented Mar 14, 2020

derekwaynecarr Mar 26, 2020

giuseppe Mar 27, 2020

derekwaynecarr left a comment

derekwaynecarr commented Mar 26, 2020

k8s-ci-robot commented Mar 26, 2020

mattjmcnaughton commented Mar 26, 2020

kubelet: add initial support for cgroupv2 #85218

kubelet: add initial support for cgroupv2 #85218

Conversation

giuseppe commented Nov 13, 2019 • edited

k8s-ci-robot commented Nov 13, 2019

mattjmcnaughton left a comment

Choose a reason for hiding this comment

mrunalp commented Nov 14, 2019

giuseppe commented Nov 18, 2019

derekwaynecarr commented Mar 12, 2020

derekwaynecarr left a comment

Choose a reason for hiding this comment

derekwaynecarr Mar 12, 2020

Choose a reason for hiding this comment

derekwaynecarr Mar 12, 2020

Choose a reason for hiding this comment

giuseppe Mar 12, 2020

Choose a reason for hiding this comment

derekwaynecarr Mar 26, 2020

Choose a reason for hiding this comment

giuseppe commented Mar 12, 2020

mattjmcnaughton commented Mar 14, 2020

derekwaynecarr Mar 26, 2020

Choose a reason for hiding this comment

giuseppe Mar 27, 2020

Choose a reason for hiding this comment

derekwaynecarr left a comment

Choose a reason for hiding this comment

derekwaynecarr commented Mar 26, 2020

k8s-ci-robot commented Mar 26, 2020

mattjmcnaughton commented Mar 26, 2020

giuseppe commented Nov 13, 2019 •

edited