add KEP for cgroups v2 support #1370

giuseppe · 2019-11-18T14:53:32Z

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

k8s-ci-robot · 2019-11-18T14:53:39Z

Welcome @giuseppe!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2019-11-18T14:53:40Z

Hi @giuseppe. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

AkihiroSuda

Let's unshare cgroupns by default

keps/sig-node/20191118-cgroups-v2.md

AkihiroSuda · 2019-11-18T15:51:47Z

Do you think we should open a separate KEP for rootless?

giuseppe · 2019-11-18T16:11:08Z

Do you think we should open a separate KEP for rootless?

yes let's keep it separate from cgroups v2, which is already touching enough components :-)

derekwaynecarr

we need to add more detail on kubelet changes where it interacts with cgroups directly

keps/sig-node/20191118-cgroups-v2.md

derekwaynecarr · 2019-11-18T17:36:14Z

keps/sig-node/20191118-cgroups-v2.md

+
+## User Stories
+
+*   The Kubelet can run on a cgroups v2 host


Clarify that kubelet will support v1 or v2

keps/sig-node/20191118-cgroups-v2.md

derekwaynecarr · 2019-11-18T17:42:06Z

keps/sig-node/20191118-cgroups-v2.md

+
+Kubelet PR: [https://github.com/kubernetes/kubernetes/pull/85218](https://github.com/kubernetes/kubernetes/pull/85218)
+
+### Option 2: Use cgroups v2 throughout the stack


For option 1, kubelet still has to write cgroups v2 for its interactions with cgroup. Please add some detail around kubelet / cm package and eviction manager.

derekwaynecarr · 2019-11-18T17:44:19Z

keps/sig-node/20191118-cgroups-v2.md

+
+## Graduation Criteria
+
+- Alpha: Basic support for running Kubernetes on a cgroups v2 host.


Worth discussing if we should add test coverage on fedora as a good candidate upstream.

derekwaynecarr · 2019-11-18T17:45:39Z

/assign

keps/sig-node/20191118-cgroups-v2.md

squeed · 2019-11-22T18:33:16Z

keps/sig-node/20191118-cgroups-v2.md

+_net_prio_ are not available with the new version.  The alternative to
+these controllers is to use eBPF.
+
+The lack of the network controllers probably affects some CNI


I don't currently know of any CNI plugins that use cgroups for network management. (My list is not exhaustive, of course...). Bandwidth limiting, if implemented, is done via a tc filter / qdisc for all of the plugins I'm aware of.

keps/sig-node/20191118-cgroups-v2.md

yujuhong · 2019-12-03T18:53:51Z

keps/sig-node/20191118-cgroups-v2.md

+
+
+With this approach cAdvisor would have to read back values from
+cgroups v2 files (already done).


So, with this option, CRI implementations will need to map the cgroup v1 fields to cgroup v2 values, right?

Could you list what each layer needs to do for each option to make it clear?

user (pod spec)

kubelet

CRI implementation

OCI runtime

the changes in the CRI implementation and OCI runtime might be implemented in a different way, deciding where to draw the line.
For CRI-O+crun, most of the logic is in the OCI runtime itself, but that is not the only way to achieve it.
For the Kubelet, the changes in this patch should be enough (at least until we have not hugetlb available): kubernetes/kubernetes#85218

In the second phase though, when cgroup v2 is fully supported through the stack there is need to change both pod specs+CRI.

can you clarify why you feel the pod spec needs to change? i see no major reason to change the pod spec or resource representation.

I was assuming we'd like to expose all/most of the cgroup v2 features.

e.g. the memory controller on cgroup v1 allows to configure:

memory.soft_limit_in_bytes

memory.limit_in_bytes

while on cgroup v2 we have:

memory.high

memory.low

memory.max

memory.min

but that is probably out of the scope as each new feature (if needed) must go through its own KEP?

@derekwaynecarr are future improvements based on what cgroup 2 offers out of scope for the current KEP?

@giuseppe future improvements for what cgroup v2 offers should be out of scope of this kep. i would keep this kep focused on kubelet is tolerant of cgroup v2 host. adding new cgroup v2 specific features to resource model would be a separate enhancement.

To address @yujuhong question, I think we are saying the following:

no change required to pod spec

kubelet cgroup manager uses v1 or v2 mode based on its introspection of sys/fs on startup

kubelet qos and pod level cgroup creation on a v2 host uses the mapping table specified

cri implementers that support cgroup v2 hosts must use a similar mapping to ensure that pod bounding cgroup values are consistent with the container cgroup values.

oci runtime spec is not required to change as cri implementer provides mapping in the transitional period.

@giuseppe agree with above?

yujuhong · 2019-12-03T19:19:11Z

/cc @Random-Liu for containerd

k8s-ci-robot · 2019-12-03T19:19:13Z

@yujuhong: GitHub didn't allow me to request PR reviews from the following users: for, containerd.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @Random-Liu for containerd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

The rootless patches has been merged into the upstream Docker since early 2019 and no longer needs to be incubated in this repo. Another reason to remove Docker from this repo is that dockershim might not be going to support cgroup v2: kubernetes/enhancements#1370 (comment) Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda · 2020-02-18T14:18:57Z

@derekwaynecarr @yujuhong @dchen1107 PTAL?

sftim · 2020-02-22T23:37:58Z

/retest

derekwaynecarr · 2020-02-26T16:09:24Z

per dawn's comment and discussion on sig-node, we can merge this kep, and update as we find issues with implementation in 1.19. there is no disagreement on need to iterate here.

see: #1370 (comment)

/approve
/lgtm

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

mrunalp · 2020-02-26T16:27:21Z

/lgtm

k8s-ci-robot · 2020-02-26T16:27:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, giuseppe, mrunalp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-node/OWNERS~~ [derekwaynecarr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hectorj2f · 2020-02-27T10:50:50Z

awesome work!

crosbymichael · 2020-08-11T18:08:00Z

LGTM :)

smileusd · 2021-05-17T07:41:32Z

keps/sig-node/20191118-cgroups-v2.md

+| period | cpu.max | y = x| period and quota are written together|
+| quota | cpu.max | y = x| period and quota are written together|
+
+**blkio controller**


Any kuberente versions and plans to support blkio controller ?

Any kuberente versions and plans to support blkio controller ?

kubernetes/enhancements#1907 the KEP is WIP.

k8s-ci-robot added the cncf-cla: yes label Nov 18, 2019

k8s-ci-robot added the needs-ok-to-test label Nov 18, 2019

k8s-ci-robot added size/L kind/kep sig/node labels Nov 18, 2019

k8s-ci-robot requested review from dchen1107 and derekwaynecarr November 18, 2019 14:53

giuseppe mentioned this pull request Nov 18, 2019

kubelet: add initial support for cgroupv2 kubernetes/kubernetes#85218

Merged

AkihiroSuda reviewed Nov 18, 2019

View reviewed changes

keps/sig-node/20191118-cgroups-v2.md Show resolved Hide resolved

keps/sig-node/20191118-cgroups-v2.md Outdated Show resolved Hide resolved

derekwaynecarr requested changes Nov 18, 2019

View reviewed changes

k8s-ci-robot assigned derekwaynecarr Nov 18, 2019

AkihiroSuda mentioned this pull request Nov 18, 2019

sig-node: Kubelet-in-UserNS, aka Rootless mode #1371

Merged

AkihiroSuda reviewed Nov 18, 2019

View reviewed changes

keps/sig-node/20191118-cgroups-v2.md Outdated Show resolved Hide resolved

giuseppe force-pushed the cgroupsv2 branch 2 times, most recently from 83faa61 to c0c871d Compare November 20, 2019 07:49

squeed reviewed Nov 22, 2019

View reviewed changes

PatrickLang reviewed Dec 3, 2019

View reviewed changes

keps/sig-node/20191118-cgroups-v2.md Outdated Show resolved Hide resolved

yujuhong reviewed Dec 3, 2019

View reviewed changes

k8s-ci-robot requested a review from Random-Liu December 3, 2019 19:19

alban mentioned this pull request Dec 4, 2019

Gardenlet gardener/gardener#1601

Merged

AkihiroSuda mentioned this pull request Dec 4, 2019

support cgroup2 containerd/containerd#3799

Merged

giuseppe force-pushed the cgroupsv2 branch 2 times, most recently from 23ddf34 to a87c972 Compare December 9, 2019 14:54

AkihiroSuda mentioned this pull request Feb 18, 2020

Remove Docker rootless-containers/usernetes#126

Merged

giuseppe force-pushed the cgroupsv2 branch from 2fc1d40 to cba0e9c Compare February 25, 2020 18:38

k8s-ci-robot added lgtm approved labels Feb 26, 2020

keps/sig-node: add KEP for cgroups v2

5b4d3d5

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the cgroupsv2 branch from cba0e9c to 5b4d3d5 Compare February 26, 2020 16:22

k8s-ci-robot removed the lgtm label Feb 26, 2020

k8s-ci-robot assigned mrunalp Feb 26, 2020

k8s-ci-robot added the lgtm label Feb 26, 2020

k8s-ci-robot merged commit 4abcc67 into kubernetes:master Feb 26, 2020

k8s-ci-robot added this to the v1.18 milestone Feb 26, 2020

AkihiroSuda mentioned this pull request Feb 27, 2020

cgroup2: privileged and cgroupns containers/podman#5341

Closed

h-vetinari mentioned this pull request Mar 7, 2020

Tracker for cgroups v2 opencontainers/runtime-spec#1002

Closed

6 tasks

yashbhutwala mentioned this pull request Apr 14, 2020

cool utils. ask some use case for clear facebookincubator/senpai#1

Closed

neolit123 mentioned this pull request May 5, 2020

Kubelet on Fedora 31+ doesn't start without kernel parameter systemd.unified_cgroup_hierarchy = 0 kubernetes/kubernetes#90710

Closed

harche mentioned this pull request May 19, 2020

Ability to use CRI-O as runtime in kube-up (on Ubuntu for now) kubernetes/kubernetes#91209

Closed

giuseppe mentioned this pull request May 28, 2020

REQUEST: New membership for giuseppe kubernetes/org#1905

Closed

6 tasks

giuseppe mentioned this pull request Jan 12, 2021

cgroup v2 #2254

Closed

4 tasks

smileusd reviewed May 17, 2021

View reviewed changes

alban mentioned this pull request May 24, 2023

Don't use hostNetwork=true or hostPID=true inspektor-gadget/inspektor-gadget#1665

Closed

3 tasks

iholder101 mentioned this pull request Apr 10, 2025

Conversion of cgroup v1 CPU shares to v2 CPU weight causes workloads to have low CPU priority kubernetes/kubernetes#131216

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add KEP for cgroups v2 support #1370

add KEP for cgroups v2 support #1370

giuseppe commented Nov 18, 2019

k8s-ci-robot commented Nov 18, 2019

k8s-ci-robot commented Nov 18, 2019

AkihiroSuda left a comment

AkihiroSuda commented Nov 18, 2019

giuseppe commented Nov 18, 2019

derekwaynecarr left a comment

derekwaynecarr Nov 18, 2019

derekwaynecarr Nov 18, 2019

derekwaynecarr Nov 18, 2019

derekwaynecarr commented Nov 18, 2019

squeed Nov 22, 2019

yujuhong Dec 3, 2019

giuseppe Dec 9, 2019

derekwaynecarr Jan 10, 2020

giuseppe Jan 13, 2020

giuseppe Jan 16, 2020

derekwaynecarr Jan 29, 2020

yujuhong commented Dec 3, 2019

k8s-ci-robot commented Dec 3, 2019

AkihiroSuda commented Feb 18, 2020

sftim commented Feb 22, 2020

derekwaynecarr commented Feb 26, 2020

mrunalp commented Feb 26, 2020

k8s-ci-robot commented Feb 26, 2020

hectorj2f commented Feb 27, 2020

crosbymichael commented Aug 11, 2020

smileusd May 17, 2021

warmchang May 17, 2021


		Kubelet PR: [https://github.com/kubernetes/kubernetes/pull/85218](https://github.com/kubernetes/kubernetes/pull/85218)

		### Option 2: Use cgroups v2 throughout the stack


		## Graduation Criteria

		- Alpha: Basic support for running Kubernetes on a cgroups v2 host.



		With this approach cAdvisor would have to read back values from
		cgroups v2 files (already done).

add KEP for cgroups v2 support #1370

add KEP for cgroups v2 support #1370

Conversation

giuseppe commented Nov 18, 2019

k8s-ci-robot commented Nov 18, 2019

k8s-ci-robot commented Nov 18, 2019

AkihiroSuda left a comment

Choose a reason for hiding this comment

AkihiroSuda commented Nov 18, 2019

giuseppe commented Nov 18, 2019

derekwaynecarr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekwaynecarr commented Nov 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yujuhong commented Dec 3, 2019

k8s-ci-robot commented Dec 3, 2019

AkihiroSuda commented Feb 18, 2020

sftim commented Feb 22, 2020

derekwaynecarr commented Feb 26, 2020

mrunalp commented Feb 26, 2020

k8s-ci-robot commented Feb 26, 2020

hectorj2f commented Feb 27, 2020

crosbymichael commented Aug 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment