Skip to content

Add option to perform syncfs after pull #9401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Dec 12, 2023
Merged

Add option to perform syncfs after pull #9401

merged 2 commits into from
Dec 12, 2023
+137 −60

Conversation

fuweid
Copy link
Member

@fuweid fuweid commented Nov 20, 2023 •
edited
Loading

It's to ensure the data integrity during unexpected power failure.

Background

Since release 1.3, in Linux system, containerD unpacks and writes files into
overlayfs snapshot directly. It doesn’t involve any mount-umount operations
so that the performance of pulling image has been improved.

As we know, the umount syscall for overlayfs will force kernel to flush
all the dirty pages into disk. Without umount syscall, the files’ data relies
on kernel’s writeback threads or filesystem's commit setting (for
instance, ext4 filesystem).

The files in committed snapshot can be loss after unexpected power failure.
However, the snapshot has been committed and the metadata also has been
fsynced. There is data inconsistency between snapshot metadata and files
in that snapshot.

We, containerd, received several issues about data loss after unexpected
power failure.

Solution

Option 1: SyncFs after unpack

Linux platform provides syncfs syscall to synchronize just the
filesystem containing a given file.

Option 2: Fsync directories recursively and fsync on regular file

The fsync doesn't support symlink/block device/char device files. We
need to use fsync the parent directory to ensure that entry is
persisted.

However, based on xfstest-dev, there is no case to ensure
fsync-on-parent can persist the special file's metadata, for example,
uid/gid, access mode.

Checkout generic/690, generic/520: Syncing parent dir can persist
symlink. But for f2fs, it needs special mount option. And it doesn't say
that uid/gid can be persisted. All the details are behind the
filesystem implementation.

NOTE: All the related test cases has _flakey_drop_and_remount in
xfstest-dev.

Based on discussion about Documenting the crash-recovery guarantees of Linux file systems,
we can't rely on Fsync-on-parent.

Option 1 is winner

This patch is using option 1.

There is test result based on test-tool.
All the networking traffic created by pull is local.

  • Image: docker.io/library/golang:1.19.4 (992 MiB)

    • Current: 5.446738579s
      • WIOS=21081, WBytes=1329741824, RIOS=79, RBytes=1197056
    • Option 1: 6.239686088s
      • WIOS=34804, WBytes=1454845952, RIOS=79, RBytes=1197056
    • Option 2: 1m30.510934813s
      • WIOS=42143, WBytes=1471397888, RIOS=82, RBytes=1209344

  • Image: docker.io/tensorflow/tensorflow:latest (1.78 GiB, ~32590 Inodes)

    • Current: 8.852718042s
      • WIOS=39417, WBytes=2412818432, RIOS=2673, RBytes=335987712
    • Option 1: 9.683387174s
      • WIOS=42767, WBytes=2431750144, RIOS=89, RBytes=1238016
    • Option 2: 1m54.302103719s
      • WIOS=54403, WBytes=2460528640, RIOS=1709, RBytes=208237568

The Option 1 will increase wios. So, the image_pull_with_sync_fs is
option in CRI plugin.

@fuweid fuweid force-pushed the v2-mode branch 2 times, most recently from b05e1b4 to 82389b4 Compare November 20, 2023 16:55
@e-minguez e-minguez mentioned this pull request Nov 20, 2023
Closed
@fuweid
Copy link
Member Author

fuweid commented Nov 21, 2023

It's based on the discussion at weekly meeting - September 14, 2023

ping @dmcgowan @mikebrow @samuelkarp @neersighted

@zhuangqh
Copy link
Contributor

just writing file with O_direct?
write through to disk without page cache

@fuweid
Copy link
Member Author

fuweid commented Nov 25, 2023

just writing file with O_direct?

Hi @zhuangqh , there is not just regular file, but also symlink, block device and directory. For special files, it's unlikely to use O_direct to do persistence.

henry118
henry118 reviewed Dec 2, 2023
View reviewed changes
diff/apply/apply_linux.go
Comment on lines +60 to +67
case sync && len(mounts) == 1 && mounts[0].Type == "bind":
defer func() {
if retErr != nil {
return
}

retErr = doSyncFs(mounts[0].Source)
}()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for my knowledge here, we need this logic because umount syscall does not flush the cache for bind mounts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on test result, yes.

The alpine image has only one layer. Based on the following test, the file cache is gone after reboot.

https://github.com/fuweid/go-dmflakey/blob/08b32bc47733b5be82ceb3e7460d250de83112f9/contrib/test/containerd/issue5854_test.go#L32

henry118
henry118 approved these changes Dec 4, 2023
View reviewed changes
Copy link
Member

@henry118 henry118 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@113xiaoji
Copy link

Mark

samuelkarp
samuelkarp approved these changes Dec 10, 2023
View reviewed changes
diff/apply/apply_darwin.go
@@ -38,6 +38,8 @@ func apply(ctx context.Context, mounts []mount.Mount, r io.Reader) error {
path := mounts[0].Source
_, err := archive.Apply(ctx, path, r, opts...)
return err

// TODO: Do we need to sync all the filesystems?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we open an issue to track this TODO?

diff/apply/apply_other.go
@@ -26,7 +26,8 @@ import (
"github.com/containerd/containerd/v2/mount"
)

func apply(ctx context.Context, mounts []mount.Mount, r io.Reader) error {
func apply(ctx context.Context, mounts []mount.Mount, r io.Reader, _sync bool) error {
// TODO: for windows, how to sync?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, let's have an issue to track. Note that _other isn't just Windows; FreeBSD will be in this codepath too (but there's no overlayfs, so there is a mount/umount operation).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. #9497 PTAL. Thanks

@samuelkarp samuelkarp enabled auto-merge December 10, 2023 08:15
@fuweid fuweid mentioned this pull request Dec 10, 2023
Open
7 tasks
@fuweid
Copy link
Member Author

fuweid commented Dec 11, 2023

Ping @dmcgowan this pull request needs your help to skip those ghost pipelines. 😂

@fuweid fuweid disabled auto-merge December 12, 2023 02:18
@fuweid
api: introduce sync_fs to diff.ApplyRequest
bd5c602 
It's flag to synchronize the underlying filesystem containing files
created during Apply.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
@fuweid
*: introduce image_pull_with_sync_fs in CRI
23278c8 
It's to ensure the data integrity during unexpected power failure.

Background:

Since release 1.3, in Linux system, containerD unpacks and writes files into
overlayfs snapshot directly. It doesn’t involve any mount-umount operations
so that the performance of pulling image has been improved.

As we know, the umount syscall for overlayfs will force kernel to flush
all the dirty pages into disk. Without umount syscall, the files’ data relies
on kernel’s writeback threads or filesystem's commit setting (for
instance, ext4 filesystem).

The files in committed snapshot can be loss after unexpected power failure.
However, the snapshot has been committed and the metadata also has been
fsynced. There is data inconsistency between snapshot metadata and files
in that snapshot.

We, containerd, received several issues about data loss after unexpected
power failure.

* containerd#5854
* containerd#3369 (comment)

Solution:

* Option 1: SyncFs after unpack

Linux platform provides [syncfs][syncfs] syscall to synchronize just the
filesystem containing a given file.

* Option 2: Fsync directories recursively and fsync on regular file

The fsync doesn't support symlink/block device/char device files. We
need to use fsync the parent directory to ensure that entry is
persisted.

However, based on [xfstest-dev][xfstest-dev], there is no case to ensure
fsync-on-parent can persist the special file's metadata, for example,
uid/gid, access mode.

Checkout [generic/690][generic/690]: Syncing parent dir can persist
symlink. But for f2fs, it needs special mount option. And it doesn't say
that uid/gid can be persisted. All the details are behind the
implemetation.

> NOTE: All the related test cases has `_flakey_drop_and_remount` in
[xfstest-dev].

Based on discussion about [Documenting the crash-recovery guarantees of Linux file systems][kernel-crash-recovery-data-integrity],
we can't rely on Fsync-on-parent.

* Option 1 is winner

This patch is using option 1.

There is test result based on [test-tool][test-tool].
All the networking traffic created by pull is local.

  * Image: docker.io/library/golang:1.19.4 (992 MiB)
    * Current: 5.446738579s
      * WIOS=21081, WBytes=1329741824, RIOS=79, RBytes=1197056
    * Option 1: 6.239686088s
      * WIOS=34804, WBytes=1454845952, RIOS=79, RBytes=1197056
    * Option 2: 1m30.510934813s
      * WIOS=42143, WBytes=1471397888, RIOS=82, RBytes=1209344

  * Image: docker.io/tensorflow/tensorflow:latest (1.78 GiB, ~32590 Inodes)
    * Current: 8.852718042s
      * WIOS=39417, WBytes=2412818432, RIOS=2673, RBytes=335987712
    * Option 1: 9.683387174s
      * WIOS=42767, WBytes=2431750144, RIOS=89, RBytes=1238016
    * Option 2: 1m54.302103719s
      * WIOS=54403, WBytes=2460528640, RIOS=1709, RBytes=208237568

The Option 1 will increase `wios`. So, the `image_pull_with_sync_fs` is
option in CRI plugin.

[syncfs]: <https://man7.org/linux/man-pages/man2/syncfs.2.html>
[xfstest-dev]: <https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git>
[generic/690]: <https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/tree/tests/generic/690?h=v2023.11.19>
[kernel-crash-recovery-data-integrity]: <https://lore.kernel.org/linux-fsdevel/1552418820-18102-1-git-send-email-jaya@cs.utexas.edu/>
[test-tool]: <https://github.com/fuweid/go-dmflakey/blob/a17fb2010db22654b3e54cf506b0dbb5ef7b33ca/contrib/syncfs/containerd/main_test.go#L51>

Signed-off-by: Wei Fu <fuweid89@gmail.com>
@dmcgowan dmcgowan added this pull request to the merge queue Dec 12, 2023
Merged via the queue into containerd:main with commit 1feb234 Dec 12, 2023
@fuweid fuweid deleted the v2-mode branch December 12, 2023 15:28
This was referenced Dec 12, 2023
Closed
Closed
Closed
Closed
@fuweid fuweid added cherry-pick/1.6.x Change to be cherry picked to release/1.6 branch cherry-pick/1.7.x Change to be cherry picked to release/1.7 branch labels Dec 12, 2023
@pendo324 pendo324 mentioned this pull request Jan 20, 2024
Merged
1 task
@EigoOda EigoOda mentioned this pull request Aug 22, 2024
Closed
@dmcgowan dmcgowan mentioned this pull request Aug 27, 2024
Merged
@xgp01 xgp01 mentioned this pull request Aug 30, 2024
Closed
@sctb512
Copy link
Member

sctb512 commented Sep 4, 2024

@fuweid Is there any plan to cherry-pick this patch to 1.6.x ? 😄

@fuweid
Copy link
Member Author

fuweid commented Sep 7, 2024

@sctb512 I don't have bandwidth to do that recently. Please feel free to file request. Thanks

@dmcgowan dmcgowan mentioned this pull request Sep 30, 2024
Merged
Mengkzhaoyun pushed a commit to open-beagle/containerd that referenced this pull request Oct 11, 2024
Merge tag 'v2.0.0-rc.5' into release-v2.0
c737fe2 
containerd 2.0.0-rc.5

Welcome to the v2.0.0-rc.5 release of containerd!
*This is a pre-release of containerd*

The first major release of containerd 2.x focuses on the continued stability of
containerd's core feature set with an easy upgrade from containerd 1.x. This
release includes the stabilization of new features added in the last 1.x release
as well as the removal of features which were deprecated in 1.x. The goal is to
support the vast community of containerd users well into the future along with
their ever increasing deployment footprints and variety of use cases.

* Add Update API for sandbox controller ([#9903](containerd/containerd#9903))
* Configure otel from env instead of config.toml ([#8970](containerd/containerd#8970))
* Enable NRI by default ([#9744](containerd/containerd#9744))
* Add PluginInfo to introspection API ([#9442](containerd/containerd#9442))
* Remove overlayfs volatile option on temp mounts ([#9555](containerd/containerd#9555))
* Expose usage of deprecated features ([#9258](containerd/containerd#9258))
* Use Intel ISA-L's igzip if available ([#9200](containerd/containerd#9200))
* Introduce top level config migration ([#9223](containerd/containerd#9223))
* Add image delete target ([#8989](containerd/containerd#8989))
* Remove `LimitNOFILE` from `containerd.service` ([#8924](containerd/containerd#8924))
* Add support for image expiration during garbage collection ([#9022](containerd/containerd#9022))
* Reduce the contention between ref lock and boltdb lock in content store ([#8792](containerd/containerd#8792))
* Remove "containerd.io/restart.logpath" label ([#8264](containerd/containerd#8264))
* Remove `aufs` snapshotter ([#8263](containerd/containerd#8263))
* Fix deadlock during NRI plugin registration ([containerd/nri#79](containerd/nri#79))
* Fix deadlock when writing to pipe blocks ([containerd/ttrpc#168](containerd/ttrpc#168))

* Generate attestation for artifacts during release ([#10543](containerd/containerd#10543))

* Use 'UserSpecifiedImage' from CRI to set the image-name annotation ([#10747](containerd/containerd#10747))
* Add support to set loopback to up ([#10238](containerd/containerd#10238))
* Add support for multiple subscribers to CRI container events ([#9661](containerd/containerd#9661))
* Enable CDI by default ([#9621](containerd/containerd#9621))
* Remove non-sandboxed CRI implementation ([#9228](containerd/containerd#9228))
* Add support for userns in stateless and stateful pods with idmap mounts (KEP-127, k8s >= 1.27) ([#8287](containerd/containerd#8287))
* Use sandboxed CRI by default ([#8994](containerd/containerd#8994))
* Implement RuntimeConfig CRI call ([#8722](containerd/containerd#8722))
* Add support for user namespaces (KEP-127) ([#8803](containerd/containerd#8803))
* Remove CRI v1alpha2 ([#8276](containerd/containerd#8276))

* Add api Go module and move all protos under api ([#10151](containerd/containerd#10151))
* Move packages based on contributing guide ([#9365](containerd/containerd#9365))
* Generalize plugin library ([#9214](containerd/containerd#9214))
* Use github.com/containerd/log ([#9086](containerd/containerd#9086))

* Support to syncfs after pull by using diff plugin ([#10284](containerd/containerd#10284))
* Skip "unknown" in image platform listing ([#10257](containerd/containerd#10257))
* Update unpacker to fetch all provided content ([#10202](containerd/containerd#10202))
* Enable Transfer service API to support plain HTTP ([#10024](containerd/containerd#10024))
* Enable Transfer service to use registry configuration directory ([#9908](containerd/containerd#9908))
* Disable the support for Schema 1 images ([#9765](containerd/containerd#9765))
* Update Transfer service to add OCI descriptors to Progress structure ([#9630](containerd/containerd#9630))
* Update import and export to allow references to missing content  ([#9554](containerd/containerd#9554))
* Add option to perform syncfs after pull ([#9401](containerd/containerd#9401))
* Add image verifier transfer service plugin system based on a binary directory ([#8493](containerd/containerd#8493))

* Implement  RuntimeStatus.features.supplemental_groups_policy from KEP-3619 ([#10410](containerd/containerd#10410))
* Add pprof to runc-shim ([#10242](containerd/containerd#10242))
* Provide runtime options in plugin info ([#10251](containerd/containerd#10251))
* Store bootstrap parameters in sandbox metadata ([#9736](containerd/containerd#9736))
* Update apparmor to allow confined runc to kill containers ([#10123](containerd/containerd#10123))
* Support vsock connection to task api ([#9738](containerd/containerd#9738))
* Update RuntimeDefault seccomp profile to disallow io_uring related syscalls ([#9320](containerd/containerd#9320))
* Switch runc shim to task service v3 and fix restore ([#9233](containerd/containerd#9233))
* Add sandboxer configuration and move sandbox controllers to plugins ([#8268](containerd/containerd#8268))
* Add annotations to CreateSandbox request ([#8960](containerd/containerd#8960))
* Add SandboxMetrics ([#8680](containerd/containerd#8680))
* Publish sandbox events ([#8602](containerd/containerd#8602))
* Remove the CriuPath field from runc's options ([#8279](containerd/containerd#8279))
* Remove `io.containerd.runtime.v1.linux` and `io.containerd.runc.v1` ([#8262](containerd/containerd#8262))

* [medium] RAPL accessible to a container [GHSA-7ww5-4wqc-m92c](GHSA-7ww5-4wqc-m92c)

* Remove `disable_cgroup` from CRI config ([#10594](containerd/containerd#10594))
* Disable the support for Schema 1 images ([#9765](containerd/containerd#9765))
* Update RuntimeDefault seccomp profile to disallow io_uring related syscalls ([#9320](containerd/containerd#9320))
* Move client to subpackage ([#9316](containerd/containerd#9316))
* Remove `LimitNOFILE` from `containerd.service` ([#8924](containerd/containerd#8924))
* Remove CRI v1alpha2 ([#8276](containerd/containerd#8276))
* Remove `io.containerd.runtime.v1.linux` and `io.containerd.runc.v1` ([#8262](containerd/containerd#8262))
* Remove "containerd.io/restart.logpath" label ([#8264](containerd/containerd#8264))
* Remove `aufs` snapshotter ([#8263](containerd/containerd#8263))

* Update warnings for deprecated CRI config fields ([#10509](containerd/containerd#10509))
* Add type alias for event Envelope ([#10279](containerd/containerd#10279))
* Postpone removal of deprecated CRI config properties ([#9966](containerd/containerd#9966))
* Deprecate go-plugin configuration option ([#9238](containerd/containerd#9238))
* CNI conf_template in CRI is no longer deprecated ([#8637](containerd/containerd#8637))

Please try out the release binaries and report any issues at
https://github.com/containerd/containerd/issues.

* Derek McGowan
* Akihiro Suda
* Maksym Pavlenko
* Wei Fu
* Phil Estes
* Sebastiaan van Stijn
* Samuel Karp
* Stefan Berger
* Kazuyoshi Kato
* Rodrigo Campos
* Danny Canter
* Abel Feng
* Akhil Mohan
* Kirtana Ashok
* Gabriel Adrian Samfira
* Austin Vazquez
* Iceber Gu
* Krisztian Litkey
* Kohei Tokunaga
* Mike Brown
* Jin Dong
* Bjorn Neergaard
* Justin Chadwell
* rongfu.leng
* James Sturtevant
* Davanum Srinivas
* Paul "TBBle" Hampson
* Henry Wang
* Brian Goff
* Enrico Weigelt
* Laura Brehm
* Marat Radchenko
* Paweł Gronowski
* Shingo Omura
* Hsing-Yu (David) Chen
* Ilya Hanov
* Cardy.Tang
* Swagat Bora
* Aditi Sharma
* Amit Barve
* Bryant Biggs
* Evan Lezar
* James Jenkins
* Jordan Liggitt
* Kay Yan
* Markus Lehtonen
* Nashwan Azhari
* Shuaiyi Zhang
* Vinayak Goyal
* helen
* Alexandru Matei
* Anthony Nandaa
* Avi Deitcher
* Charity Kathure
* Cory Snider
* Ed Bartosh
* Etienne Champetier
* Kevin Parsons
* Michael Zappa
* Milas Bowman
* ningmingxiao
* yanggang
* zounengren
* Aditya Ramani
* Adrian Reber
* Amir M. Ghazanfari
* Artem Khramov
* Brad Davidson
* Chen Yiyang
* Christian Muehlhaeuser
* Djordje Lukic
* Edgar Lee
* Eric Lin
* Ethan Lowman
* Jiang Liu
* June Rhodes
* Kern Walster
* Lucas Rattz
* Mahamed Ali
* Maksim An
* Michael Crosby
* Peteris Rudzusiks
* Sam Edwards
* Samruddhi Khandale
* Sascha Grunert
* Steve Griffith
* Tony Fang
* VERNOU Cédric
* Vishal Reddy Gurrala
* hang.jiang
* harshitasao
* jerryzhuang
* lengrongfu
* roman-kiselenko
* zhanluxianshen
* Aaron Lehmann
* Adrien Delorme
* Alex Couture-Beil
* Alex Ellis
* Alex Rodriguez
* Angelos Kolaitis
* Antonio Huete Jimenez
* Arash Haghighat
* Ben Foster
* Bin Tang
* Bin Xin
* BinBin He
* Brennan Kinney
* Changqing Li
* ChengenH
* ChengyuZhu6
* Christian Stewart
* Colin O'Dell
* Craig Ingram
* Daisy Rong
* David Porter
* Derek Nola
* Eng Zer Jun
* Erikson Tung
* Fabiano Fidêncio
* Fahed Dorgaa
* Gary McDonald
* Iain Macdonald
* James Lakin
* Jan Dubois
* Jaroslav Jindrak
* Javier Maestro
* Jian Wang
* Jiongchi Yu
* Julien Balestra
* Kir Kolyshkin
* Kirill A. Korinsky
* Konstantin Khlebnikov
* Mauri de Souza Meneguzzo
* Pan Yibo
* Paul Meyer
* Qasim Sarfraz
* Qiutong Song
* Reinhard Tartler
* Robbie Buxton
* Robert-André Mauchin
* Ruihua Wen
* Sameer
* Shengjing Zhu
* Shiming Zhang
* Shukui Yang
* Talon
* Tariq Ibrahim
* Tianon Gravi
* Tim Hockin
* TinaMor
* Tobias Klauser
* Tomáš Virtus
* Tõnis Tiigi
* Wang Xinwen
* William Chen
* Xinyang Ge
* Yibo Zhuang
* Yury Gargay
* Zechun Chen
* Zhang Tianyang
* Zoe
* baijia
* bo.jiang
* bzsuni
* charles-chenzz
* chschumacher1994
* guangli.bao
* guangwu
* jinda.ljd
* krglosse
* pigletfly
* rokkiter
* wangxiang
* zhangpeng
* zhaojizhuang
* 吴小白
* 张钰
* 沈陵
* 谭九鼎

* **dario.cat/mergo**                                                              v1.0.1 **_new_**
* **github.com/AdaLogics/go-fuzz-headers**                                         1f10f66a31bf -> ced1acdcaa24
* **github.com/AdamKorcz/go-118-fuzz-build**                                       5330a85ea652 -> 8075edf89bb0
* **github.com/Microsoft/go-winio**                                                v0.6.0 -> v0.6.2
* **github.com/Microsoft/hcsshim**                                                 v0.10.0-rc.7 -> v0.12.6
* **github.com/cenkalti/backoff/v4**                                               v4.2.0 -> v4.3.0
* **github.com/cespare/xxhash/v2**                                                 v2.2.0 -> v2.3.0
* **github.com/checkpoint-restore/checkpointctl**                                  v1.2.1 **_new_**
* **github.com/checkpoint-restore/go-criu/v7**                                     v7.2.0 **_new_**
* **github.com/cilium/ebpf**                                                       v0.9.1 -> v0.11.0
* **github.com/containerd/cgroups/v3**                                             v3.0.1 -> v3.0.3
* **github.com/containerd/console**                                                v1.0.3 -> v1.0.4
* **github.com/containerd/containerd/api**                                         v1.8.0-rc.3 **_new_**
* **github.com/containerd/continuity**                                             v0.3.0 -> v0.4.3
* **github.com/containerd/errdefs**                                                v0.1.0 **_new_**
* **github.com/containerd/go-cni**                                                 v1.1.9 -> v1.1.10
* **github.com/containerd/go-runc**                                                v1.0.0 -> v1.1.0
* **github.com/containerd/imgcrypt**                                               v1.1.7 -> v1.2.0-rc1
* **github.com/containerd/log**                                                    v0.1.0 **_new_**
* **github.com/containerd/nri**                                                    v0.3.0 -> v0.6.1
* **github.com/containerd/otelttrpc**                                              ea5083fda723 **_new_**
* **github.com/containerd/platforms**                                              v0.2.1 **_new_**
* **github.com/containerd/plugin**                                                 v0.1.0 **_new_**
* **github.com/containerd/ttrpc**                                                  v1.2.1 -> v1.2.5
* **github.com/containerd/typeurl/v2**                                             v2.1.0 -> v2.2.0
* **github.com/containernetworking/cni**                                           v1.1.2 -> v1.2.3
* **github.com/containernetworking/plugins**                                       v1.2.0 -> v1.5.1
* **github.com/containers/ocicrypt**                                               v1.1.6 -> v1.2.0
* **github.com/cpuguy83/go-md2man/v2**                                             v2.0.2 -> v2.0.4
* **github.com/davecgh/go-spew**                                                   v1.1.1 -> d8f796af33cc
* **github.com/distribution/reference**                                            v0.6.0 **_new_**
* **github.com/emicklei/go-restful/v3**                                            v3.10.1 -> v3.11.0
* **github.com/felixge/httpsnoop**                                                 v1.0.4 **_new_**
* **github.com/fsnotify/fsnotify**                                                 v1.6.0 -> v1.7.0
* **github.com/fxamacker/cbor/v2**                                                 v2.7.0 **_new_**
* **github.com/go-jose/go-jose/v4**                                                v4.0.2 **_new_**
* **github.com/go-logr/logr**                                                      v1.2.3 -> v1.4.2
* **github.com/golang/protobuf**                                                   v1.5.2 -> v1.5.4
* **github.com/google/go-cmp**                                                     v0.5.9 -> v0.6.0
* **github.com/google/uuid**                                                       v1.3.0 -> v1.6.0
* **github.com/gorilla/websocket**                                                 v1.5.0 **_new_**
* **github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus**            v1.0.1 **_new_**
* **github.com/grpc-ecosystem/go-grpc-middleware/v2**                              v2.1.0 **_new_**
* **github.com/grpc-ecosystem/grpc-gateway/v2**                                    v2.7.0 -> v2.22.0
* **github.com/intel/goresctrl**                                                   v0.3.0 -> v0.7.0
* **github.com/klauspost/compress**                                                v1.16.0 -> v1.17.10
* **github.com/mdlayher/socket**                                                   v0.4.1 **_new_**
* **github.com/mdlayher/vsock**                                                    v1.2.1 **_new_**
* **github.com/moby/spdystream**                                                   v0.2.0 -> v0.4.0
* **github.com/moby/sys/mountinfo**                                                v0.6.2 -> v0.7.2
* **github.com/moby/sys/sequential**                                               v0.5.0 -> v0.6.0
* **github.com/moby/sys/signal**                                                   v0.7.0 -> v0.7.1
* **github.com/moby/sys/symlink**                                                  v0.2.0 -> v0.3.0
* **github.com/moby/sys/user**                                                     v0.3.0 **_new_**
* **github.com/moby/sys/userns**                                                   v0.1.0 **_new_**
* **github.com/munnerz/goautoneg**                                                 a7dc8b61c822 **_new_**
* **github.com/mxk/go-flowrate**                                                   cca7078d478f **_new_**
* **github.com/opencontainers/image-spec**                                         3a7f492d3f1b -> v1.1.0
* **github.com/opencontainers/runtime-spec**                                       v1.1.0-rc.1 -> v1.2.0
* **github.com/opencontainers/runtime-tools**                                      946c877fa809 -> 2e043c6bd626
* **github.com/pelletier/go-toml/v2**                                              v2.2.3 **_new_**
* **github.com/pmezard/go-difflib**                                                v1.0.0 -> 5d4384ee4fb2
* **github.com/prometheus/client_golang**                                          v1.14.0 -> v1.20.4
* **github.com/prometheus/client_model**                                           v0.3.0 -> v0.6.1
* **github.com/prometheus/common**                                                 v0.37.0 -> v0.55.0
* **github.com/prometheus/procfs**                                                 v0.8.0 -> v0.15.1
* **github.com/sirupsen/logrus**                                                   v1.9.0 -> v1.9.3
* **github.com/stretchr/testify**                                                  v1.8.2 -> v1.9.0
* **github.com/urfave/cli/v2**                                                     v2.27.4 **_new_**
* **github.com/vishvananda/netlink**                                               v1.2.1-beta.2 -> v1.3.0
* **github.com/vishvananda/netns**                                                 2eb08e3e575f -> v0.0.4
* **github.com/x448/float16**                                                      v0.8.4 **_new_**
* **github.com/xrash/smetrics**                                                    686a1a2994c1 **_new_**
* **go.etcd.io/bbolt**                                                             v1.3.7 -> v1.3.11
* **go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc**  v0.40.0 -> v0.55.0
* **go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp**                v0.55.0 **_new_**
* **go.opentelemetry.io/otel**                                                     v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace**                            v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc**              v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp**              v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/metric**                                              v0.37.0 -> v1.30.0
* **go.opentelemetry.io/otel/sdk**                                                 v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/trace**                                               v1.14.0 -> v1.30.0
* **go.opentelemetry.io/proto/otlp**                                               v0.19.0 -> v1.3.1
* **golang.org/x/crypto**                                                          v0.1.0 -> v0.27.0
* **golang.org/x/exp**                                                             aacd6d4b4611 **_new_**
* **golang.org/x/mod**                                                             v0.7.0 -> v0.21.0
* **golang.org/x/net**                                                             v0.7.0 -> v0.29.0
* **golang.org/x/oauth2**                                                          v0.4.0 -> v0.22.0
* **golang.org/x/sync**                                                            v0.1.0 -> v0.8.0
* **golang.org/x/sys**                                                             v0.6.0 -> v0.25.0
* **golang.org/x/term**                                                            v0.5.0 -> v0.24.0
* **golang.org/x/text**                                                            v0.7.0 -> v0.18.0
* **golang.org/x/time**                                                            90d013bbcef8 -> v0.3.0
* **google.golang.org/genproto/googleapis/api**                                    8af14fe29dc1 **_new_**
* **google.golang.org/genproto/googleapis/rpc**                                    8af14fe29dc1 **_new_**
* **google.golang.org/grpc**                                                       v1.53.0 -> v1.67.0
* **google.golang.org/protobuf**                                                   v1.28.1 -> v1.34.2
* **k8s.io/api**                                                                   v0.26.2 -> v0.31.1
* **k8s.io/apimachinery**                                                          v0.26.2 -> v0.31.1
* **k8s.io/apiserver**                                                             v0.26.2 -> v0.31.1
* **k8s.io/client-go**                                                             v0.26.2 -> v0.31.1
* **k8s.io/component-base**                                                        v0.26.2 -> v0.31.1
* **k8s.io/cri-api**                                                               v0.26.2 -> v0.32.0-alpha.0
* **k8s.io/klog/v2**                                                               v2.90.1 -> v2.130.1
* **k8s.io/kubelet**                                                               v0.31.1 **_new_**
* **k8s.io/utils**                                                                 a5ecb0141aa5 -> 18e509b52bc8
* **sigs.k8s.io/json**                                                             f223a00ba0e2 -> bc3834ca7abd
* **sigs.k8s.io/structured-merge-diff/v4**                                         v4.2.3 -> v4.4.1
* **sigs.k8s.io/yaml**                                                             v1.3.0 -> v1.4.0
* **tags.cncf.io/container-device-interface**                                      v0.8.0 **_new_**
* **tags.cncf.io/container-device-interface/specs-go**                             v0.8.0 **_new_**

Previous release can be found at [v1.7.0](https://github.com/containerd/containerd/releases/tag/v1.7.0)
* `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`:         ✅Recommended. Dynamically linked with glibc 2.31 (Ubuntu 20.04).
* `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`:  Statically linked. Expected to be used on non-glibc Linux distributions. Not position-independent.

In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases)
and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too.

See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.
This was referenced Oct 17, 2024
Open
Open
kiashok added a commit to kiashok/containerd-containerd that referenced this pull request Oct 23, 2024
@kiashok
Merge tag 'v2.0.0-rc.5' into dev/kiashok/update-2.0-rc5
cf261aa 
containerd 2.0.0-rc.5

Welcome to the v2.0.0-rc.5 release of containerd!
*This is a pre-release of containerd*

The first major release of containerd 2.x focuses on the continued stability of
containerd's core feature set with an easy upgrade from containerd 1.x. This
release includes the stabilization of new features added in the last 1.x release
as well as the removal of features which were deprecated in 1.x. The goal is to
support the vast community of containerd users well into the future along with
their ever increasing deployment footprints and variety of use cases.

* Add Update API for sandbox controller ([containerd#9903](containerd#9903))
* Configure otel from env instead of config.toml ([containerd#8970](containerd#8970))
* Enable NRI by default ([containerd#9744](containerd#9744))
* Add PluginInfo to introspection API ([containerd#9442](containerd#9442))
* Remove overlayfs volatile option on temp mounts ([containerd#9555](containerd#9555))
* Expose usage of deprecated features ([containerd#9258](containerd#9258))
* Use Intel ISA-L's igzip if available ([containerd#9200](containerd#9200))
* Introduce top level config migration ([containerd#9223](containerd#9223))
* Add image delete target ([containerd#8989](containerd#8989))
* Remove `LimitNOFILE` from `containerd.service` ([containerd#8924](containerd#8924))
* Add support for image expiration during garbage collection ([containerd#9022](containerd#9022))
* Reduce the contention between ref lock and boltdb lock in content store ([containerd#8792](containerd#8792))
* Remove "containerd.io/restart.logpath" label ([containerd#8264](containerd#8264))
* Remove `aufs` snapshotter ([containerd#8263](containerd#8263))
* Fix deadlock during NRI plugin registration ([containerd/nri#79](containerd/nri#79))
* Fix deadlock when writing to pipe blocks ([containerd/ttrpc#168](containerd/ttrpc#168))

* Generate attestation for artifacts during release ([containerd#10543](containerd#10543))

* Use 'UserSpecifiedImage' from CRI to set the image-name annotation ([containerd#10747](containerd#10747))
* Add support to set loopback to up ([containerd#10238](containerd#10238))
* Add support for multiple subscribers to CRI container events ([containerd#9661](containerd#9661))
* Enable CDI by default ([containerd#9621](containerd#9621))
* Remove non-sandboxed CRI implementation ([containerd#9228](containerd#9228))
* Add support for userns in stateless and stateful pods with idmap mounts (KEP-127, k8s >= 1.27) ([containerd#8287](containerd#8287))
* Use sandboxed CRI by default ([containerd#8994](containerd#8994))
* Implement RuntimeConfig CRI call ([containerd#8722](containerd#8722))
* Add support for user namespaces (KEP-127) ([containerd#8803](containerd#8803))
* Remove CRI v1alpha2 ([containerd#8276](containerd#8276))

* Add api Go module and move all protos under api ([containerd#10151](containerd#10151))
* Move packages based on contributing guide ([containerd#9365](containerd#9365))
* Generalize plugin library ([containerd#9214](containerd#9214))
* Use github.com/containerd/log ([containerd#9086](containerd#9086))

* Support to syncfs after pull by using diff plugin ([containerd#10284](containerd#10284))
* Skip "unknown" in image platform listing ([containerd#10257](containerd#10257))
* Update unpacker to fetch all provided content ([containerd#10202](containerd#10202))
* Enable Transfer service API to support plain HTTP ([containerd#10024](containerd#10024))
* Enable Transfer service to use registry configuration directory ([containerd#9908](containerd#9908))
* Disable the support for Schema 1 images ([containerd#9765](containerd#9765))
* Update Transfer service to add OCI descriptors to Progress structure ([containerd#9630](containerd#9630))
* Update import and export to allow references to missing content  ([containerd#9554](containerd#9554))
* Add option to perform syncfs after pull ([containerd#9401](containerd#9401))
* Add image verifier transfer service plugin system based on a binary directory ([containerd#8493](containerd#8493))

* Implement  RuntimeStatus.features.supplemental_groups_policy from KEP-3619 ([containerd#10410](containerd#10410))
* Add pprof to runc-shim ([containerd#10242](containerd#10242))
* Provide runtime options in plugin info ([containerd#10251](containerd#10251))
* Store bootstrap parameters in sandbox metadata ([containerd#9736](containerd#9736))
* Update apparmor to allow confined runc to kill containers ([containerd#10123](containerd#10123))
* Support vsock connection to task api ([containerd#9738](containerd#9738))
* Update RuntimeDefault seccomp profile to disallow io_uring related syscalls ([containerd#9320](containerd#9320))
* Switch runc shim to task service v3 and fix restore ([containerd#9233](containerd#9233))
* Add sandboxer configuration and move sandbox controllers to plugins ([containerd#8268](containerd#8268))
* Add annotations to CreateSandbox request ([containerd#8960](containerd#8960))
* Add SandboxMetrics ([containerd#8680](containerd#8680))
* Publish sandbox events ([containerd#8602](containerd#8602))
* Remove the CriuPath field from runc's options ([containerd#8279](containerd#8279))
* Remove `io.containerd.runtime.v1.linux` and `io.containerd.runc.v1` ([containerd#8262](containerd#8262))

* [medium] RAPL accessible to a container [GHSA-7ww5-4wqc-m92c](GHSA-7ww5-4wqc-m92c)

* Remove `disable_cgroup` from CRI config ([containerd#10594](containerd#10594))
* Disable the support for Schema 1 images ([containerd#9765](containerd#9765))
* Update RuntimeDefault seccomp profile to disallow io_uring related syscalls ([containerd#9320](containerd#9320))
* Move client to subpackage ([containerd#9316](containerd#9316))
* Remove `LimitNOFILE` from `containerd.service` ([containerd#8924](containerd#8924))
* Remove CRI v1alpha2 ([containerd#8276](containerd#8276))
* Remove `io.containerd.runtime.v1.linux` and `io.containerd.runc.v1` ([containerd#8262](containerd#8262))
* Remove "containerd.io/restart.logpath" label ([containerd#8264](containerd#8264))
* Remove `aufs` snapshotter ([containerd#8263](containerd#8263))

* Update warnings for deprecated CRI config fields ([containerd#10509](containerd#10509))
* Add type alias for event Envelope ([containerd#10279](containerd#10279))
* Postpone removal of deprecated CRI config properties ([containerd#9966](containerd#9966))
* Deprecate go-plugin configuration option ([containerd#9238](containerd#9238))
* CNI conf_template in CRI is no longer deprecated ([containerd#8637](containerd#8637))

Please try out the release binaries and report any issues at
https://github.com/containerd/containerd/issues.

* Derek McGowan
* Akihiro Suda
* Maksym Pavlenko
* Wei Fu
* Phil Estes
* Sebastiaan van Stijn
* Samuel Karp
* Stefan Berger
* Kazuyoshi Kato
* Rodrigo Campos
* Danny Canter
* Abel Feng
* Akhil Mohan
* Kirtana Ashok
* Gabriel Adrian Samfira
* Austin Vazquez
* Iceber Gu
* Krisztian Litkey
* Kohei Tokunaga
* Mike Brown
* Jin Dong
* Bjorn Neergaard
* Justin Chadwell
* rongfu.leng
* James Sturtevant
* Davanum Srinivas
* Paul "TBBle" Hampson
* Henry Wang
* Brian Goff
* Enrico Weigelt
* Laura Brehm
* Marat Radchenko
* Paweł Gronowski
* Shingo Omura
* Hsing-Yu (David) Chen
* Ilya Hanov
* Cardy.Tang
* Swagat Bora
* Aditi Sharma
* Amit Barve
* Bryant Biggs
* Evan Lezar
* James Jenkins
* Jordan Liggitt
* Kay Yan
* Markus Lehtonen
* Nashwan Azhari
* Shuaiyi Zhang
* Vinayak Goyal
* helen
* Alexandru Matei
* Anthony Nandaa
* Avi Deitcher
* Charity Kathure
* Cory Snider
* Ed Bartosh
* Etienne Champetier
* Kevin Parsons
* Michael Zappa
* Milas Bowman
* ningmingxiao
* yanggang
* zounengren
* Aditya Ramani
* Adrian Reber
* Amir M. Ghazanfari
* Artem Khramov
* Brad Davidson
* Chen Yiyang
* Christian Muehlhaeuser
* Djordje Lukic
* Edgar Lee
* Eric Lin
* Ethan Lowman
* Jiang Liu
* June Rhodes
* Kern Walster
* Lucas Rattz
* Mahamed Ali
* Maksim An
* Michael Crosby
* Peteris Rudzusiks
* Sam Edwards
* Samruddhi Khandale
* Sascha Grunert
* Steve Griffith
* Tony Fang
* VERNOU Cédric
* Vishal Reddy Gurrala
* hang.jiang
* harshitasao
* jerryzhuang
* lengrongfu
* roman-kiselenko
* zhanluxianshen
* Aaron Lehmann
* Adrien Delorme
* Alex Couture-Beil
* Alex Ellis
* Alex Rodriguez
* Angelos Kolaitis
* Antonio Huete Jimenez
* Arash Haghighat
* Ben Foster
* Bin Tang
* Bin Xin
* BinBin He
* Brennan Kinney
* Changqing Li
* ChengenH
* ChengyuZhu6
* Christian Stewart
* Colin O'Dell
* Craig Ingram
* Daisy Rong
* David Porter
* Derek Nola
* Eng Zer Jun
* Erikson Tung
* Fabiano Fidêncio
* Fahed Dorgaa
* Gary McDonald
* Iain Macdonald
* James Lakin
* Jan Dubois
* Jaroslav Jindrak
* Javier Maestro
* Jian Wang
* Jiongchi Yu
* Julien Balestra
* Kir Kolyshkin
* Kirill A. Korinsky
* Konstantin Khlebnikov
* Mauri de Souza Meneguzzo
* Pan Yibo
* Paul Meyer
* Qasim Sarfraz
* Qiutong Song
* Reinhard Tartler
* Robbie Buxton
* Robert-André Mauchin
* Ruihua Wen
* Sameer
* Shengjing Zhu
* Shiming Zhang
* Shukui Yang
* Talon
* Tariq Ibrahim
* Tianon Gravi
* Tim Hockin
* TinaMor
* Tobias Klauser
* Tomáš Virtus
* Tõnis Tiigi
* Wang Xinwen
* William Chen
* Xinyang Ge
* Yibo Zhuang
* Yury Gargay
* Zechun Chen
* Zhang Tianyang
* Zoe
* baijia
* bo.jiang
* bzsuni
* charles-chenzz
* chschumacher1994
* guangli.bao
* guangwu
* jinda.ljd
* krglosse
* pigletfly
* rokkiter
* wangxiang
* zhangpeng
* zhaojizhuang
* 吴小白
* 张钰
* 沈陵
* 谭九鼎

* **dario.cat/mergo**                                                              v1.0.1 **_new_**
* **github.com/AdaLogics/go-fuzz-headers**                                         1f10f66a31bf -> ced1acdcaa24
* **github.com/AdamKorcz/go-118-fuzz-build**                                       5330a85ea652 -> 8075edf89bb0
* **github.com/Microsoft/go-winio**                                                v0.6.0 -> v0.6.2
* **github.com/Microsoft/hcsshim**                                                 v0.10.0-rc.7 -> v0.12.6
* **github.com/cenkalti/backoff/v4**                                               v4.2.0 -> v4.3.0
* **github.com/cespare/xxhash/v2**                                                 v2.2.0 -> v2.3.0
* **github.com/checkpoint-restore/checkpointctl**                                  v1.2.1 **_new_**
* **github.com/checkpoint-restore/go-criu/v7**                                     v7.2.0 **_new_**
* **github.com/cilium/ebpf**                                                       v0.9.1 -> v0.11.0
* **github.com/containerd/cgroups/v3**                                             v3.0.1 -> v3.0.3
* **github.com/containerd/console**                                                v1.0.3 -> v1.0.4
* **github.com/containerd/containerd/api**                                         v1.8.0-rc.3 **_new_**
* **github.com/containerd/continuity**                                             v0.3.0 -> v0.4.3
* **github.com/containerd/errdefs**                                                v0.1.0 **_new_**
* **github.com/containerd/go-cni**                                                 v1.1.9 -> v1.1.10
* **github.com/containerd/go-runc**                                                v1.0.0 -> v1.1.0
* **github.com/containerd/imgcrypt**                                               v1.1.7 -> v1.2.0-rc1
* **github.com/containerd/log**                                                    v0.1.0 **_new_**
* **github.com/containerd/nri**                                                    v0.3.0 -> v0.6.1
* **github.com/containerd/otelttrpc**                                              ea5083fda723 **_new_**
* **github.com/containerd/platforms**                                              v0.2.1 **_new_**
* **github.com/containerd/plugin**                                                 v0.1.0 **_new_**
* **github.com/containerd/ttrpc**                                                  v1.2.1 -> v1.2.5
* **github.com/containerd/typeurl/v2**                                             v2.1.0 -> v2.2.0
* **github.com/containernetworking/cni**                                           v1.1.2 -> v1.2.3
* **github.com/containernetworking/plugins**                                       v1.2.0 -> v1.5.1
* **github.com/containers/ocicrypt**                                               v1.1.6 -> v1.2.0
* **github.com/cpuguy83/go-md2man/v2**                                             v2.0.2 -> v2.0.4
* **github.com/davecgh/go-spew**                                                   v1.1.1 -> d8f796af33cc
* **github.com/distribution/reference**                                            v0.6.0 **_new_**
* **github.com/emicklei/go-restful/v3**                                            v3.10.1 -> v3.11.0
* **github.com/felixge/httpsnoop**                                                 v1.0.4 **_new_**
* **github.com/fsnotify/fsnotify**                                                 v1.6.0 -> v1.7.0
* **github.com/fxamacker/cbor/v2**                                                 v2.7.0 **_new_**
* **github.com/go-jose/go-jose/v4**                                                v4.0.2 **_new_**
* **github.com/go-logr/logr**                                                      v1.2.3 -> v1.4.2
* **github.com/golang/protobuf**                                                   v1.5.2 -> v1.5.4
* **github.com/google/go-cmp**                                                     v0.5.9 -> v0.6.0
* **github.com/google/uuid**                                                       v1.3.0 -> v1.6.0
* **github.com/gorilla/websocket**                                                 v1.5.0 **_new_**
* **github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus**            v1.0.1 **_new_**
* **github.com/grpc-ecosystem/go-grpc-middleware/v2**                              v2.1.0 **_new_**
* **github.com/grpc-ecosystem/grpc-gateway/v2**                                    v2.7.0 -> v2.22.0
* **github.com/intel/goresctrl**                                                   v0.3.0 -> v0.7.0
* **github.com/klauspost/compress**                                                v1.16.0 -> v1.17.10
* **github.com/mdlayher/socket**                                                   v0.4.1 **_new_**
* **github.com/mdlayher/vsock**                                                    v1.2.1 **_new_**
* **github.com/moby/spdystream**                                                   v0.2.0 -> v0.4.0
* **github.com/moby/sys/mountinfo**                                                v0.6.2 -> v0.7.2
* **github.com/moby/sys/sequential**                                               v0.5.0 -> v0.6.0
* **github.com/moby/sys/signal**                                                   v0.7.0 -> v0.7.1
* **github.com/moby/sys/symlink**                                                  v0.2.0 -> v0.3.0
* **github.com/moby/sys/user**                                                     v0.3.0 **_new_**
* **github.com/moby/sys/userns**                                                   v0.1.0 **_new_**
* **github.com/munnerz/goautoneg**                                                 a7dc8b61c822 **_new_**
* **github.com/mxk/go-flowrate**                                                   cca7078d478f **_new_**
* **github.com/opencontainers/image-spec**                                         3a7f492d3f1b -> v1.1.0
* **github.com/opencontainers/runtime-spec**                                       v1.1.0-rc.1 -> v1.2.0
* **github.com/opencontainers/runtime-tools**                                      946c877fa809 -> 2e043c6bd626
* **github.com/pelletier/go-toml/v2**                                              v2.2.3 **_new_**
* **github.com/pmezard/go-difflib**                                                v1.0.0 -> 5d4384ee4fb2
* **github.com/prometheus/client_golang**                                          v1.14.0 -> v1.20.4
* **github.com/prometheus/client_model**                                           v0.3.0 -> v0.6.1
* **github.com/prometheus/common**                                                 v0.37.0 -> v0.55.0
* **github.com/prometheus/procfs**                                                 v0.8.0 -> v0.15.1
* **github.com/sirupsen/logrus**                                                   v1.9.0 -> v1.9.3
* **github.com/stretchr/testify**                                                  v1.8.2 -> v1.9.0
* **github.com/urfave/cli/v2**                                                     v2.27.4 **_new_**
* **github.com/vishvananda/netlink**                                               v1.2.1-beta.2 -> v1.3.0
* **github.com/vishvananda/netns**                                                 2eb08e3e575f -> v0.0.4
* **github.com/x448/float16**                                                      v0.8.4 **_new_**
* **github.com/xrash/smetrics**                                                    686a1a2994c1 **_new_**
* **go.etcd.io/bbolt**                                                             v1.3.7 -> v1.3.11
* **go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc**  v0.40.0 -> v0.55.0
* **go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp**                v0.55.0 **_new_**
* **go.opentelemetry.io/otel**                                                     v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace**                            v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc**              v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp**              v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/metric**                                              v0.37.0 -> v1.30.0
* **go.opentelemetry.io/otel/sdk**                                                 v1.14.0 -> v1.30.0
* **go.opentelemetry.io/otel/trace**                                               v1.14.0 -> v1.30.0
* **go.opentelemetry.io/proto/otlp**                                               v0.19.0 -> v1.3.1
* **golang.org/x/crypto**                                                          v0.1.0 -> v0.27.0
* **golang.org/x/exp**                                                             aacd6d4b4611 **_new_**
* **golang.org/x/mod**                                                             v0.7.0 -> v0.21.0
* **golang.org/x/net**                                                             v0.7.0 -> v0.29.0
* **golang.org/x/oauth2**                                                          v0.4.0 -> v0.22.0
* **golang.org/x/sync**                                                            v0.1.0 -> v0.8.0
* **golang.org/x/sys**                                                             v0.6.0 -> v0.25.0
* **golang.org/x/term**                                                            v0.5.0 -> v0.24.0
* **golang.org/x/text**                                                            v0.7.0 -> v0.18.0
* **golang.org/x/time**                                                            90d013bbcef8 -> v0.3.0
* **google.golang.org/genproto/googleapis/api**                                    8af14fe29dc1 **_new_**
* **google.golang.org/genproto/googleapis/rpc**                                    8af14fe29dc1 **_new_**
* **google.golang.org/grpc**                                                       v1.53.0 -> v1.67.0
* **google.golang.org/protobuf**                                                   v1.28.1 -> v1.34.2
* **k8s.io/api**                                                                   v0.26.2 -> v0.31.1
* **k8s.io/apimachinery**                                                          v0.26.2 -> v0.31.1
* **k8s.io/apiserver**                                                             v0.26.2 -> v0.31.1
* **k8s.io/client-go**                                                             v0.26.2 -> v0.31.1
* **k8s.io/component-base**                                                        v0.26.2 -> v0.31.1
* **k8s.io/cri-api**                                                               v0.26.2 -> v0.32.0-alpha.0
* **k8s.io/klog/v2**                                                               v2.90.1 -> v2.130.1
* **k8s.io/kubelet**                                                               v0.31.1 **_new_**
* **k8s.io/utils**                                                                 a5ecb0141aa5 -> 18e509b52bc8
* **sigs.k8s.io/json**                                                             f223a00ba0e2 -> bc3834ca7abd
* **sigs.k8s.io/structured-merge-diff/v4**                                         v4.2.3 -> v4.4.1
* **sigs.k8s.io/yaml**                                                             v1.3.0 -> v1.4.0
* **tags.cncf.io/container-device-interface**                                      v0.8.0 **_new_**
* **tags.cncf.io/container-device-interface/specs-go**                             v0.8.0 **_new_**

Previous release can be found at [v1.7.0](https://github.com/containerd/containerd/releases/tag/v1.7.0)
* `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`:         ✅Recommended. Dynamically linked with glibc 2.31 (Ubuntu 20.04).
* `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`:  Statically linked. Expected to be used on non-glibc Linux distributions. Not position-independent.

In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases)
and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too.

See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.
@dmcgowan dmcgowan mentioned this pull request Oct 23, 2024
Merged
@dmcgowan dmcgowan mentioned this pull request Nov 4, 2024
Merged
@github-actions github-actions bot mentioned this pull request Nov 10, 2024
Open
Mengkzhaoyun pushed a commit to open-beagle/containerd that referenced this pull request Nov 11, 2024
Merge tag 'v2.0.0' into release-v2.0
2f44e98 
containerd 2.0.0

Welcome to the v2.0.0 release of containerd!

The first major release of containerd 2.x focuses on the continued stability of
containerd's core feature set with an easy upgrade from containerd 1.x. This
release includes the stabilization of new features added in the last 1.x release
as well as the removal of features which were deprecated in 1.x. The goal is to
support the vast community of containerd users well into the future along with
their ever increasing deployment footprints and variety of use cases.

See [containerd 2.0](https://github.com/containerd/containerd/blob/main/docs/containerd-2.0.md) documentation for details on what is new and has changed in this release.

* Allow sections of Plugins to be merged, and not overwritten as entire sections. ([#9982](containerd/containerd#9982))
* Add Update API for sandbox controller ([#9903](containerd/containerd#9903))
* Configure otel from env instead of config.toml ([#8970](containerd/containerd#8970))
* Enable NRI by default ([#9744](containerd/containerd#9744))
* Add PluginInfo to introspection API ([#9442](containerd/containerd#9442))
* Remove overlayfs volatile option on temp mounts ([#9555](containerd/containerd#9555))
* Expose usage of deprecated features ([#9258](containerd/containerd#9258))
* Use Intel ISA-L's igzip if available ([#9200](containerd/containerd#9200))
* Introduce top level config migration ([#9223](containerd/containerd#9223))
* Add image delete target ([#8989](containerd/containerd#8989))
* Remove `LimitNOFILE` from `containerd.service` ([#8924](containerd/containerd#8924))
* Add support for image expiration during garbage collection ([#9022](containerd/containerd#9022))
* Reduce the contention between ref lock and boltdb lock in content store ([#8792](containerd/containerd#8792))
* Remove "containerd.io/restart.logpath" label ([#8264](containerd/containerd#8264))
* Remove `aufs` snapshotter ([#8263](containerd/containerd#8263))
* Fix deadlock during NRI plugin registration ([containerd/nri#79](containerd/nri#79))
* Support arm64/v9 and minor variants ([containerd/platforms#8](containerd/platforms#8))
* Fix deadlock when writing to pipe blocks ([containerd/ttrpc#168](containerd/ttrpc#168))

* Generate attestation for artifacts during release ([#10543](containerd/containerd#10543))
* Remove `cri-containerd-*.tar.gz` release bundles ([#9096](containerd/containerd#9096))

* Use 'UserSpecifiedImage' from CRI to set the image-name annotation ([#10747](containerd/containerd#10747))
* Fine-grained SupplementalGroups control ([#9737](containerd/containerd#9737))
* Add support to set loopback to up ([#10238](containerd/containerd#10238))
* KEP-3857: Recursive Read-only (RRO) mounts ([#9787](containerd/containerd#9787))
* Add support for multiple subscribers to CRI container events ([#9661](containerd/containerd#9661))
* Enable CDI by default ([#9621](containerd/containerd#9621))
* Remove non-sandboxed CRI implementation ([#9228](containerd/containerd#9228))
* Add support for userns in stateless and stateful pods with idmap mounts (KEP-127, k8s >= 1.27) ([#8287](containerd/containerd#8287))
* Use sandboxed CRI by default ([#8994](containerd/containerd#8994))
* Implement RuntimeConfig CRI call ([#8722](containerd/containerd#8722))
* Add support for user namespaces (KEP-127) ([#8803](containerd/containerd#8803))
* Remove CRI v1alpha2 ([#8276](containerd/containerd#8276))

* Add api Go module and move all protos under api ([#10151](containerd/containerd#10151))
* Move packages based on contributing guide ([#9365](containerd/containerd#9365))
* Generalize plugin library ([#9214](containerd/containerd#9214))
* Use github.com/containerd/log ([#9086](containerd/containerd#9086))

* Support to syncfs after pull by using diff plugin ([#10284](containerd/containerd#10284))
* Skip "unknown" in image platform listing ([#10257](containerd/containerd#10257))
* Update unpacker to fetch all provided content ([#10202](containerd/containerd#10202))
* Enable Transfer service API to support plain HTTP ([#10024](containerd/containerd#10024))
* Enable Transfer service to use registry configuration directory ([#9908](containerd/containerd#9908))
* Disable the support for Schema 1 images ([#9765](containerd/containerd#9765))
* Update Transfer service to add OCI descriptors to Progress structure ([#9630](containerd/containerd#9630))
* Update import and export to allow references to missing content  ([#9554](containerd/containerd#9554))
* Add option to perform syncfs after pull ([#9401](containerd/containerd#9401))
* Add image verifier transfer service plugin system based on a binary directory ([#8493](containerd/containerd#8493))

* Implement  RuntimeStatus.features.supplemental_groups_policy from KEP-3619 ([#10410](containerd/containerd#10410))
* Add pprof to runc-shim ([#10242](containerd/containerd#10242))
* Provide runtime options in plugin info ([#10251](containerd/containerd#10251))
* Store bootstrap parameters in sandbox metadata ([#9736](containerd/containerd#9736))
* Update apparmor to allow confined runc to kill containers ([#10123](containerd/containerd#10123))
* Support vsock connection to task api ([#9738](containerd/containerd#9738))
* Update RuntimeDefault seccomp profile to disallow io_uring related syscalls ([#9320](containerd/containerd#9320))
* Switch runc shim to task service v3 and fix restore ([#9233](containerd/containerd#9233))
* Add sandboxer configuration and move sandbox controllers to plugins ([#8268](containerd/containerd#8268))
* Add annotations to CreateSandbox request ([#8960](containerd/containerd#8960))
* Add SandboxMetrics ([#8680](containerd/containerd#8680))
* Publish sandbox events ([#8602](containerd/containerd#8602))
* Remove the CriuPath field from runc's options ([#8279](containerd/containerd#8279))
* Remove `io.containerd.runtime.v1.linux` and `io.containerd.runc.v1` ([#8262](containerd/containerd#8262))

* [medium] RAPL accessible to a container [GHSA-7ww5-4wqc-m92c](GHSA-7ww5-4wqc-m92c)

* Remove `disable_cgroup` from CRI config ([#10594](containerd/containerd#10594))
* Disable the support for Schema 1 images ([#9765](containerd/containerd#9765))
* Update RuntimeDefault seccomp profile to disallow io_uring related syscalls ([#9320](containerd/containerd#9320))
* Move client to subpackage ([#9316](containerd/containerd#9316))
* Remove `LimitNOFILE` from `containerd.service` ([#8924](containerd/containerd#8924))
* Remove CRI v1alpha2 ([#8276](containerd/containerd#8276))
* Remove `io.containerd.runtime.v1.linux` and `io.containerd.runc.v1` ([#8262](containerd/containerd#8262))
* Remove "containerd.io/restart.logpath" label ([#8264](containerd/containerd#8264))
* Remove `aufs` snapshotter ([#8263](containerd/containerd#8263))

* Update warnings for deprecated CRI config fields ([#10509](containerd/containerd#10509))
* Add type alias for event Envelope ([#10279](containerd/containerd#10279))
* Postpone removal of deprecated CRI config properties ([#9966](containerd/containerd#9966))
* Deprecate go-plugin configuration option ([#9238](containerd/containerd#9238))
* CNI conf_template in CRI is no longer deprecated ([#8637](containerd/containerd#8637))

Please try out the release binaries and report any issues at
https://github.com/containerd/containerd/issues.

* Derek McGowan
* Akihiro Suda
* Maksym Pavlenko
* Wei Fu
* Phil Estes
* Sebastiaan van Stijn
* Samuel Karp
* Krisztian Litkey
* Kazuyoshi Kato
* Austin Vazquez
* Rodrigo Campos
* Danny Canter
* Abel Feng
* Mike Brown
* Kirtana Ashok
* Akhil Mohan
* Iceber Gu
* Gabriel Adrian Samfira
* Jin Dong
* Kohei Tokunaga
* Bjorn Neergaard
* Brian Goff
* Justin Chadwell
* rongfu.leng
* James Sturtevant
* Davanum Srinivas
* Paul "TBBle" Hampson
* Henry Wang
* Enrico Weigelt
* Laura Brehm
* Marat Radchenko
* Paweł Gronowski
* Shingo Omura
* Hsing-Yu (David) Chen
* Ilya Hanov
* Cardy.Tang
* Swagat Bora
* Aditi Sharma
* Amit Barve
* Bryant Biggs
* Evan Lezar
* James Jenkins
* Jordan Liggitt
* Kay Yan
* Markus Lehtonen
* Nashwan Azhari
* Shuaiyi Zhang
* Vinayak Goyal
* helen
* Alexandru Matei
* Anthony Nandaa
* Avi Deitcher
* Charity Kathure
* Cory Snider
* Ed Bartosh
* Etienne Champetier
* Kevin Parsons
* Michael Zappa
* Milas Bowman
* lengrongfu
* ningmingxiao
* yanggang
* zounengren
* Aditya Ramani
* Adrian Reber
* Amir M. Ghazanfari
* Antonio Ojea
* Artem Khramov
* Brad Davidson
* Chen Yiyang
* Chongyi Zheng
* Christian Muehlhaeuser
* Djordje Lukic
* Edgar Lee
* Eric Lin
* Ethan Lowman
* Jiang Liu
* June Rhodes
* Kern Walster
* Lei Jitang
* Lucas Rattz
* Mahamed Ali
* Maksim An
* Michael Crosby
* Peteris Rudzusiks
* Ray Burgemeestre
* Sam Edwards
* Samruddhi Khandale
* Sascha Grunert
* Steve Griffith
* Tony Fang
* Tõnis Tiigi
* VERNOU Cédric
* Vishal Reddy Gurrala
* Xiaojin Zhang
* Yang Yang
* hang.jiang
* harshitasao
* jerryzhuang
* roman-kiselenko
* zhanluxianshen
* Aaron Lehmann
* AbdelrahmanElawady
* Adrien Delorme
* Alex Couture-Beil
* Alex Ellis
* Alex Rodriguez
* Angelos Kolaitis
* Antonio Huete Jimenez
* Antti Kervinen
* Arash Haghighat
* Arkin Modi
* Ben Foster
* Benjamin Peterson
* Bin Tang
* Bin Xin
* BinBin He
* Brennan Kinney
* Changqing Li
* ChengenH
* ChengyuZhu6
* Christian Stewart
* Colin O'Dell
* Craig Ingram
* Daisy Rong
* David Porter
* David Son
* Derek Nola
* Eng Zer Jun
* Erikson Tung
* Fabiano Fidêncio
* Fahed Dorgaa
* Gabriela Cervantes
* Gary McDonald
* Iain Macdonald
* James Lakin
* Jan Dubois
* Jaroslav Jindrak
* Javier Maestro
* Jian Wang
* Jiongchi Yu
* Julien Balestra
* Kir Kolyshkin
* Kirill A. Korinsky
* Konstantin Khlebnikov
* Lei Liu
* Matteo Pulcini
* Mauri de Souza Meneguzzo
* Mike Baynton
* Niklas Gehlen
* Pan Yibo
* Paul Meyer
* Qasim Sarfraz
* Qiutong Song
* Reinhard Tartler
* Robbie Buxton
* Robert-André Mauchin
* Ruihua Wen
* Saket Jajoo
* Sameer
* Shengjing Zhu
* Shiming Zhang
* Shukui Yang
* StepSecurity Bot
* Talon
* Tariq Ibrahim
* Tianon Gravi
* Tim Hockin
* TinaMor
* Tobias Klauser
* Tomáš Virtus
* Wang Xinwen
* William Chen
* Xinyang Ge
* Yibo Zhuang
* Yuhang Wei
* Yury Gargay
* Zechun Chen
* Zhang Tianyang
* Zoe
* baijia
* bo.jiang
* bzsuni
* charles-chenzz
* chschumacher1994
* cormick
* guangli.bao
* guangwu
* jinda.ljd
* jingtao.liang
* krglosse
* pigletfly
* rokkiter
* wangxiang
* zhangpeng
* zhaojizhuang
* 吴小白
* 张钰
* 沈陵
* 谭九鼎

* **dario.cat/mergo**                                                              v1.0.1 **_new_**
* **github.com/AdaLogics/go-fuzz-headers**                                         1f10f66a31bf -> e8a1dd7889d6
* **github.com/AdamKorcz/go-118-fuzz-build**                                       5330a85ea652 -> 2b5cbb29f3e2
* **github.com/Microsoft/go-winio**                                                v0.6.0 -> v0.6.2
* **github.com/Microsoft/hcsshim**                                                 v0.10.0-rc.7 -> v0.12.9
* **github.com/cenkalti/backoff/v4**                                               v4.2.0 -> v4.3.0
* **github.com/cespare/xxhash/v2**                                                 v2.2.0 -> v2.3.0
* **github.com/checkpoint-restore/checkpointctl**                                  v1.3.0 **_new_**
* **github.com/checkpoint-restore/go-criu/v7**                                     v7.2.0 **_new_**
* **github.com/cilium/ebpf**                                                       v0.9.1 -> v0.11.0
* **github.com/containerd/cgroups/v3**                                             v3.0.1 -> v3.0.3
* **github.com/containerd/console**                                                v1.0.3 -> v1.0.4
* **github.com/containerd/containerd/api**                                         v1.8.0 **_new_**
* **github.com/containerd/continuity**                                             v0.3.0 -> v0.4.4
* **github.com/containerd/errdefs**                                                v1.0.0 **_new_**
* **github.com/containerd/errdefs/pkg**                                            v0.3.0 **_new_**
* **github.com/containerd/go-cni**                                                 v1.1.9 -> v1.1.10
* **github.com/containerd/go-runc**                                                v1.0.0 -> v1.1.0
* **github.com/containerd/imgcrypt/v2**                                            v2.0.0-rc.1 **_new_**
* **github.com/containerd/log**                                                    v0.1.0 **_new_**
* **github.com/containerd/nri**                                                    v0.3.0 -> v0.8.0
* **github.com/containerd/otelttrpc**                                              ea5083fda723 **_new_**
* **github.com/containerd/platforms**                                              v1.0.0-rc.0 **_new_**
* **github.com/containerd/plugin**                                                 v1.0.0 **_new_**
* **github.com/containerd/ttrpc**                                                  v1.2.1 -> v1.2.6
* **github.com/containerd/typeurl/v2**                                             v2.1.0 -> v2.2.2
* **github.com/containerd/zfs/v2**                                                 v2.0.0-rc.0 **_new_**
* **github.com/containernetworking/cni**                                           v1.1.2 -> v1.2.3
* **github.com/containernetworking/plugins**                                       v1.2.0 -> v1.5.1
* **github.com/containers/ocicrypt**                                               v1.1.6 -> v1.2.0
* **github.com/cpuguy83/go-md2man/v2**                                             v2.0.2 -> v2.0.5
* **github.com/davecgh/go-spew**                                                   v1.1.1 -> d8f796af33cc
* **github.com/distribution/reference**                                            v0.6.0 **_new_**
* **github.com/emicklei/go-restful/v3**                                            v3.10.1 -> v3.11.0
* **github.com/felixge/httpsnoop**                                                 v1.0.4 **_new_**
* **github.com/fsnotify/fsnotify**                                                 v1.6.0 -> v1.7.0
* **github.com/fxamacker/cbor/v2**                                                 v2.7.0 **_new_**
* **github.com/go-jose/go-jose/v4**                                                v4.0.4 **_new_**
* **github.com/go-logr/logr**                                                      v1.2.3 -> v1.4.2
* **github.com/golang/protobuf**                                                   v1.5.2 -> v1.5.4
* **github.com/google/go-cmp**                                                     v0.5.9 -> v0.6.0
* **github.com/google/uuid**                                                       v1.3.0 -> v1.6.0
* **github.com/gorilla/websocket**                                                 v1.5.0 **_new_**
* **github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus**            v1.0.1 **_new_**
* **github.com/grpc-ecosystem/go-grpc-middleware/v2**                              v2.1.0 **_new_**
* **github.com/grpc-ecosystem/grpc-gateway/v2**                                    v2.7.0 -> v2.22.0
* **github.com/intel/goresctrl**                                                   v0.3.0 -> v0.8.0
* **github.com/klauspost/compress**                                                v1.16.0 -> v1.17.11
* **github.com/mdlayher/socket**                                                   v0.4.1 **_new_**
* **github.com/mdlayher/vsock**                                                    v1.2.1 **_new_**
* **github.com/mistifyio/go-zfs/v3**                                               v3.0.1 **_new_**
* **github.com/moby/spdystream**                                                   v0.2.0 -> v0.4.0
* **github.com/moby/sys/mountinfo**                                                v0.6.2 -> v0.7.2
* **github.com/moby/sys/sequential**                                               v0.5.0 -> v0.6.0
* **github.com/moby/sys/signal**                                                   v0.7.0 -> v0.7.1
* **github.com/moby/sys/symlink**                                                  v0.2.0 -> v0.3.0
* **github.com/moby/sys/user**                                                     v0.3.0 **_new_**
* **github.com/moby/sys/userns**                                                   v0.1.0 **_new_**
* **github.com/munnerz/goautoneg**                                                 a7dc8b61c822 **_new_**
* **github.com/mxk/go-flowrate**                                                   cca7078d478f **_new_**
* **github.com/opencontainers/image-spec**                                         3a7f492d3f1b -> v1.1.0
* **github.com/opencontainers/runtime-spec**                                       v1.1.0-rc.1 -> v1.2.0
* **github.com/opencontainers/runtime-tools**                                      946c877fa809 -> 2e043c6bd626
* **github.com/opencontainers/selinux**                                            v1.11.0 -> v1.11.1
* **github.com/pelletier/go-toml/v2**                                              v2.2.3 **_new_**
* **github.com/pmezard/go-difflib**                                                v1.0.0 -> 5d4384ee4fb2
* **github.com/prometheus/client_golang**                                          v1.14.0 -> v1.20.5
* **github.com/prometheus/client_model**                                           v0.3.0 -> v0.6.1
* **github.com/prometheus/common**                                                 v0.37.0 -> v0.55.0
* **github.com/prometheus/procfs**                                                 v0.8.0 -> v0.15.1
* **github.com/sirupsen/logrus**                                                   v1.9.0 -> v1.9.3
* **github.com/stefanberger/go-pkcs11uri**                                         78d3cae3a980 -> 78284954bff6
* **github.com/stretchr/testify**                                                  v1.8.2 -> v1.9.0
* **github.com/urfave/cli/v2**                                                     v2.27.5 **_new_**
* **github.com/vishvananda/netlink**                                               v1.2.1-beta.2 -> v1.3.0
* **github.com/vishvananda/netns**                                                 2eb08e3e575f -> v0.0.4
* **github.com/x448/float16**                                                      v0.8.4 **_new_**
* **github.com/xrash/smetrics**                                                    686a1a2994c1 **_new_**
* **go.etcd.io/bbolt**                                                             v1.3.7 -> v1.3.11
* **go.mozilla.org/pkcs7**                                                         432b2356ecb1 -> v0.9.0
* **go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc**  v0.40.0 -> v0.56.0
* **go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp**                v0.56.0 **_new_**
* **go.opentelemetry.io/otel**                                                     v1.14.0 -> v1.31.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace**                            v1.14.0 -> v1.31.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc**              v1.14.0 -> v1.31.0
* **go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp**              v1.14.0 -> v1.31.0
* **go.opentelemetry.io/otel/metric**                                              v0.37.0 -> v1.31.0
* **go.opentelemetry.io/otel/sdk**                                                 v1.14.0 -> v1.31.0
* **go.opentelemetry.io/otel/trace**                                               v1.14.0 -> v1.31.0
* **go.opentelemetry.io/proto/otlp**                                               v0.19.0 -> v1.3.1
* **golang.org/x/crypto**                                                          v0.1.0 -> v0.28.0
* **golang.org/x/exp**                                                             aacd6d4b4611 **_new_**
* **golang.org/x/mod**                                                             v0.7.0 -> v0.21.0
* **golang.org/x/net**                                                             v0.7.0 -> v0.30.0
* **golang.org/x/oauth2**                                                          v0.4.0 -> v0.22.0
* **golang.org/x/sync**                                                            v0.1.0 -> v0.8.0
* **golang.org/x/sys**                                                             v0.6.0 -> v0.26.0
* **golang.org/x/term**                                                            v0.5.0 -> v0.25.0
* **golang.org/x/text**                                                            v0.7.0 -> v0.19.0
* **golang.org/x/time**                                                            90d013bbcef8 -> v0.3.0
* **google.golang.org/genproto/googleapis/api**                                    5fefd90f89a9 **_new_**
* **google.golang.org/genproto/googleapis/rpc**                                    324edc3d5d38 **_new_**
* **google.golang.org/grpc**                                                       v1.53.0 -> v1.67.1
* **google.golang.org/protobuf**                                                   v1.28.1 -> v1.35.1
* **k8s.io/api**                                                                   v0.26.2 -> v0.31.2
* **k8s.io/apimachinery**                                                          v0.26.2 -> v0.31.2
* **k8s.io/apiserver**                                                             v0.26.2 -> v0.31.2
* **k8s.io/client-go**                                                             v0.26.2 -> v0.31.2
* **k8s.io/component-base**                                                        v0.26.2 -> v0.31.2
* **k8s.io/cri-api**                                                               v0.26.2 -> v0.31.2
* **k8s.io/klog/v2**                                                               v2.90.1 -> v2.130.1
* **k8s.io/kubelet**                                                               v0.31.2 **_new_**
* **k8s.io/utils**                                                                 a5ecb0141aa5 -> 18e509b52bc8
* **sigs.k8s.io/json**                                                             f223a00ba0e2 -> bc3834ca7abd
* **sigs.k8s.io/structured-merge-diff/v4**                                         v4.2.3 -> v4.4.1
* **sigs.k8s.io/yaml**                                                             v1.3.0 -> v1.4.0
* **tags.cncf.io/container-device-interface**                                      v0.8.0 **_new_**
* **tags.cncf.io/container-device-interface/specs-go**                             v0.8.0 **_new_**

Previous release can be found at [v1.7.0](https://github.com/containerd/containerd/releases/tag/v1.7.0)
* `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`:         ✅Recommended. Dynamically linked with glibc 2.31 (Ubuntu 20.04).
* `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`:  Statically linked. Expected to be used on non-glibc Linux distributions. Not position-independent.

In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases)
and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too.

See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.
@tboevil
Copy link
Contributor

tboevil commented Nov 28, 2024

Sorry to bother you. I used this script to set image_pull_with_sync_fs to true, but the final test result is still /bin/busybox: exec format error. Is there something wrong with my understanding of this MR?

@mikebrow
Copy link
Member

Sorry to bother you. I used this script to set image_pull_with_sync_fs to true, but the final test result is still /bin/busybox: exec format error. Is there something wrong with my understanding of this MR?

What script? What MR? After changing the config.toml restart containerd. The test you linked is supposed to fail in order for it to pass.

@tboevil
Copy link
Contributor

tboevil commented Nov 29, 2024 •
edited
Loading

What script? What MR? After changing the config.toml restart containerd. The test you linked is supposed to fail in order for it to pass.

Sorry, I didn't make it clear, I changed ctrdCmd := runContainerd(t, rootfs, "", logPath) to

cfgPath := filepath.Join(t.TempDir(), "config.toml")
require.NoError(t,
    os.WriteFile(cfgPath, []byte(`
version = 2

[plugins]
  [plugins.'io.containerd.grpc.v1.cri']
    image_pull_with_sync_fs = true
`),
			0600),
	)
ctrdCmd := runContainerd(t, rootfs, cfgPath, logPath)

But the final execution result is still bin/busybox: exec format error. This means the test still passes. I thought that MR would ensure the write success after setting image_pull_with_sync_fs = true.

@tboevil
Copy link
Contributor

tboevil commented Nov 29, 2024 

func TestIssue5854(t *testing.T) {
	// NOTE: Set commit=1000 is to ensure that the global writeback won't
	// persist all the data during the simulation of power failure. So,
	// if the process doesn't call fsync/fdatasync, the data won't be
	// committed into disk.
	//
	// REF:
	// Query about ext4 commit interval vs dirty_expire_centisecs - https://lore.kernel.org/linux-ext4/20191213155912.GH15474@quack2.suse.cz/
	flakey := testutils.InitFlakeyDevice(t, t.Name(), dmflakey.FSTypeEXT4, "commit=1000")

	rootfs := flakey.RootFS()

	logPath := filepath.Join(t.TempDir(), "containerd.log")
	sockAddr := filepath.Join(rootfs, "run", "containerd", "containerd.sock")

	cfgPath := filepath.Join(t.TempDir(), "config.toml")
	require.NoError(t,
		os.WriteFile(cfgPath, []byte(`
version = 2

[plugins]
  [plugins.'io.containerd.grpc.v1.cri']
    image_pull_with_sync_fs = true
`),
			0600),
	)
	
	t.Log("Start to run containerd")
	ctrdCmd := runContainerd(t, rootfs, cfgPath, logPath)

	t.Log("Wait for ready")
	expectLog(t, logPath, "containerd successfully booted in")

	imageName := "ghcr.io/containerd/alpine:3.14.0"
	t.Logf("Pulling %s", imageName)
	t.Log(sockAddr)
	runCommand(t, "ctr", "-a", sockAddr, "i","pull", imageName)

	debugCmd := exec.Command("ctr", "-a", sockAddr, "i", "ls")
	out1, err := debugCmd.CombinedOutput()
	t.Logf("before: %s", out1)
	
	t.Log("Power failure")
	ctrdCmd.Process.Kill()
	require.Error(t, ctrdCmd.Wait())
	require.NoError(t, flakey.PowerFailure(""))

	t.Log("Restarting containerd")
	ctrdCmd = runContainerd(t, rootfs, "", logPath)

	t.Log("Wait for ready")
	expectLog(t, logPath, "containerd successfully booted in")
	defer ctrdCmd.Process.Kill()

	targetMount := filepath.Join(t.TempDir())
	defer sysutils.UnmountAll(targetMount, 0)

	t.Logf("Mounting image %s on %s", imageName, targetMount)
	debugCmd2 := exec.Command("ctr", "-a", sockAddr, "i", "ls")
	out2, err := debugCmd2.CombinedOutput()
	t.Logf("after: %s", out2)
	
	runCommand(t, "ctr", "-a", sockAddr, "image", "mount", imageName, targetMount)

	t.Logf("Run busybox (Chroot: %s)", targetMount)

	debugCmd3 := exec.Command("ls", "-al", path.Join(targetMount, "/bin"))
	out3, err := debugCmd3.CombinedOutput()
	t.Logf("ls -al `targetMount`: %s", out3)

	busyboxCmd := exec.Command(filepath.Join("/bin", "busybox"))
	busyboxCmd.SysProcAttr = &syscall.SysProcAttr{Chroot: targetMount}
	_, err = busyboxCmd.CombinedOutput()
	

	require.ErrorContains(t, err, "/bin/busybox: exec format error")
	t.Log(err.Error())
}
[root@cloudtower ~]# ./main -test.v
=== RUN   TestIssue5854
    issue5854_test.go:60: Start to run containerd
    issue5854_test.go:63: Wait for ready
    issue5854_test.go:67: Pulling ghcr.io/containerd/alpine:3.14.0
    issue5854_test.go:68: /tmp/TestIssue58541552633806/002/run/containerd/containerd.sock
    issue5854_test.go:73: before: REF                              TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS
        ghcr.io/containerd/alpine:3.14.0 application/vnd.docker.distribution.manifest.list.v2+json sha256:adab3844f497ab9171f070d4cae4114b5aec565ac772e2f2579405b78be67c96 2.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
    issue5854_test.go:75: Power failure
    issue5854_test.go:80: Restarting containerd
    issue5854_test.go:83: Wait for ready
    issue5854_test.go:90: Mounting image ghcr.io/containerd/alpine:3.14.0 on /tmp/TestIssue58541552633806/005
    issue5854_test.go:93: after: REF                              TYPE                                                      DIGEST                                                                  SIZE    PLATFORMS                                                                                LABELS
        ghcr.io/containerd/alpine:3.14.0 application/vnd.docker.distribution.manifest.list.v2+json sha256:adab3844f497ab9171f070d4cae4114b5aec565ac772e2f2579405b78be67c96 2.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
    issue5854_test.go:97: Run busybox (Chroot: /tmp/TestIssue58541552633806/005)
    issue5854_test.go:104: ls -al `targetMount`: total 8
        drwxr-xr-x  2 root root 4096 Jun 15  2021 .
        drwxr-xr-x 19 root root 4096 Nov 29 10:38 ..
        lrwxrwxrwx  1 root root   12 Jun 15  2021 arch -> /bin/busybox
        lrwxrwxrwx  1 root root   12 Jun 15  2021 ash -> /bin/busybox
        lrwxrwxrwx  1 root root   12 Jun 15  2021 base64 -> /bin/busybox
        lrwxrwxrwx  1 root root   12 Jun 15  2021 bbconfig -> /bin/busybox
        -rwxr-xr-x  1 root root    0 Jun 14  2021 busybox
        ......
    issue5854_test.go:109: fork/exec /bin/busybox: exec format error
--- PASS: TestIssue5854 (7.76s)

The complete code and results are shown above

@fuweid
Copy link
Member Author

fuweid commented Dec 3, 2024

@tboevil Would you please show containerd version? just want to ensure that the containerd binary supports this flag thanks.

@tboevil
Copy link
Contributor

tboevil commented Dec 4, 2024 •
edited
Loading

@tboevil Would you please show containerd version? just want to ensure that the containerd binary supports this flag thanks.

[root@xxx ~]# ctr -v
ctr containerd.io 1.7.23
[root@xxx ~]# runc -v
runc version 1.1.14
commit: v1.1.14-0-g2c9f560
spec: 1.0.2-dev
go: go1.22.8
libseccomp: 2.3.1

@fuweid
Copy link
Member Author

fuweid commented Dec 4, 2024

Hi @tboevil What is the result of containerd --version?

@tboevil
Copy link
Contributor

tboevil commented Dec 4, 2024 •
edited
Loading

Hi @tboevil What is the result of containerd --version?

[root@xxx ~]# containerd --version
containerd containerd.io 1.7.23 57f17b0a6295a39009d861b89e3b3b87b005ca27

@fuweid
Copy link
Member Author

fuweid commented Dec 4, 2024

@tboevil I revisited your code and found that you're using ctr instead of crictl. The image_pull_with_sync_fs is for crictl instead of ctr. ctr doesn't read containerd config. Please check this code https://github.com/fuweid/go-dmflakey/blob/08b32bc47733b5be82ceb3e7460d250de83112f9/contrib/test/containerd/issue5854_test.go#L32. Hope it can help you. thanks

@tboevil
Copy link
Contributor

tboevil commented Dec 6, 2024 •
edited
Loading

@fuweid Thank you very much，I think I found the problem.

When I used the above code to test, I found that specifying the socket with -r did not work, so I changed it to ctr. Now I found the reason, because my /etc/crictl.yaml configured image-endpoint, so the -r in the test code did not work. After I changed it to -i, everything worked fine. Thanks again.

But I have another question I want to confirm. Is it possible that the problem fixed by this MR will occur when using ctr load?

@fuweid
Copy link
Member Author

fuweid commented Dec 6, 2024 •
edited
Loading

But I have another question I want to confirm. Is it possible that the problem fixed by this MR will occur when using ctr load?

Hi @tboevil

One option is to file new pull request to add option in ctr to use syncfs.
Or backport this pull request #10284 to release/1.7 and force containerd to syncfs every single time.

I suggest first option for ctr. ctr is debug tool. It's ok to add this feature.

REF: https://samuel.karp.dev/blog/2024/12/containerd-internals-ctr/

@tboevil tboevil mentioned this pull request Dec 9, 2024
Merged
@tboevil
Copy link
Contributor

tboevil commented Dec 9, 2024

Hi @tboevil

One option is to file new pull request to add option in ctr to use syncfs. Or backport this pull request #10284 to release/1.7 and force containerd to syncfs every single time.

I suggest first option for ctr. ctr is debug tool. It's ok to add this feature.

REF: https://samuel.karp.dev/blog/2024/12/containerd-internals-ctr/

Hi @fuweid
I have submitted a PR for review.

I also believe that cherry-picking this commit to version 1.7 would be beneficial, because it is an optional feature and does not update the user’s default values. If needed, I can help with the cherry-pick.

@ryan-beisner
Copy link

@tboevil +1 to cherry-picking this to 1.7.x. For k8s users experiencing difficulties with the exec fmt errors, waiting for 2.x will be quite a ways off into the future.

@shaolingxie
Copy link

hi @fuweid
I added ExecStopPost=sync to contianerd.service, thinking it would work similarly to fssync. However, the sync operation doesn’t seem to take effect, and the issue persists. What am I misunderstanding here? (containerd version is 1.6.x)
Clipboard_Screenshot_1740023967

@fuweid
Copy link
Member Author

fuweid commented Feb 21, 2025

Hi @shaolingxie

ExecStopPost won't happen when kernel is panic or power outage. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dmcgowan dmcgowan

@samuelkarp samuelkarp

@henry118 henry118

Assignees
No one assigned
Labels
area/distribution Image Distribution cherry-pick/1.6.x Change to be cherry picked to release/1.6 branch cherry-picked/1.7.x PR commits are cherry-picked into release/1.7 branch impact/changelog size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet