Remove kube-proxy userspace modes #112133

knabben · 2022-08-30T20:59:13Z

What type of PR is this?

/sig network
/sig windows
/kind cleanup
/kind api-change

What this PR does / why we need it:

Remove both Linux and Windows userspace deprecated Kube-proxy modes.
Remove the UDPIdleTimeout config used only on this mode.

Which issue(s) this PR fixes:

Fixes #103860

Special notes for your reviewer:

A few points:

UDPIdleTimeout config is gone since only used on userspace
Both fallbacks do not exist anymore, so we need to agree on what will happen, fail close when it is not supported, maybe ensuring iptables and kernel space checks as a first thing?

Does this PR introduce a user-facing change?

kube-proxy: The `userspace` proxy mode (deprecated for over a year) is no longer supported on either Linux or Windows.  Users should use `iptables` or `ipvs` on Linux, or `kernelspace` on Windows.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

danwinship · 2022-08-31T14:30:50Z

Both fallbacks do not exist anymore, so we need to agree on what will happen, fail close when it is not supported, maybe ensuring iptables and kernel space checks as a first thing?

All systems that could have run the userspace can also run the iptables proxy.

There is no longer any fallback-ing in the Linux proxy (#111806). So if someone is specifying --mode userspace, the behavior that is consistent with the rest of kube-proxy's current behavior would be to fail with an error. I'm not sure that's consistent with our deprecation rules though. (ie, this would make 1.25→1.26 upgrades fail catastrophically for anyone still using --mode userspace.) So we might want to just treat that as "warn and then fall back to iptables".

danwinship · 2022-08-31T14:32:46Z

cmd/kube-proxy/app/server.go

@@ -160,7 +160,7 @@ func (o *Options) AddFlags(fs *pflag.FlagSet) {
 	fs.Var(&utilflag.IPPortVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.")
 	fs.BoolVar(&o.config.BindAddressHardFail, "bind-address-hard-fail", o.config.BindAddressHardFail, "If true kube-proxy will treat failure to bind to a port as fatal and exit")
 	fs.Var(utilflag.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.")
-	fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'iptables' (Linux-only), 'ipvs' (Linux-only), 'kernelspace' (Windows-only), or 'userspace' (Linux/Windows, deprecated). The default value is 'iptables' on Linux and 'userspace' on Windows(will be 'kernelspace' in a future release). "+
+	fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'iptables' (Linux-only), 'ipvs' (Linux-only), 'kernelspace' (Windows-only). The default value is 'iptables' on Linux and 'kernelspace' on Windows. "+


maybe rewrite this some more now that there are no overlapping Linux/Windows modes?

"Which proxy mode to use: on Linux this can be 'iptables' (default) or 'ipvs'. On Windows the only supported value is 'kernelspace'."

?

danwinship · 2022-08-31T14:37:24Z

cmd/kube-proxy/app/server_others.go

+	// Always ordered as IPv4, IPv6
+	if primaryProtocol == utiliptables.ProtocolIPv4 {
+		ipt[0] = iptInterface
+		ipt[1] = utiliptables.New(execer, utiliptables.ProtocolIPv6)


The iptinterface initialization stuff could be rewritten to make more sense now. eg, you can remove iptInterface (and ProxyServer.IptInterface) and only have ipt, and you don't even need to figure out primaryProtocol unless dualStack gets set false

(Maybe do this as an additional refactoring commit to make it more obvious that this commit is just removing userspace and nothign else.)

(Oh, I see you left a "todo" about EndpointSlices so maybe you're planning on doing another PR later?)

@danwinship I would prefer to refactor this part on a separated commit and let this interface as is for now, wdyt?

On this PR already cleaned the UseEndpointSlices and forced the modes to use it by default.

danwinship · 2022-08-31T14:38:52Z

cmd/kube-proxy/app/server_others.go

+	for _, perFamilyIpt := range ipt {
+		if !perFamilyIpt.Present() {
+			klog.V(0).InfoS("kube-proxy running in single-stack mode, this ipFamily is not supported", "ipFamily", perFamilyIpt.Protocol())
+			dualStack = false


(lol, this doesn't actually check that primaryProtocol is the one that's supported...)

danwinship · 2022-08-31T14:40:22Z

cmd/kube-proxy/app/server_others.go

@@ -367,7 +338,7 @@ func newProxyServer(
 		OOMScoreAdj:            config.OOMScoreAdj,
 		ConfigSyncPeriod:       config.ConfigSyncPeriod.Duration,
 		HealthzServer:          healthzServer,
-		UseEndpointSlices:      useEndpointSlices,
+		UseEndpointSlices:      true, // todo (knabben) change all places


TODO or FIXME is more standard

My intent is to remove the options and keep the true code path, going to fix the todo yet on this PR.

danwinship · 2022-08-31T14:41:34Z

cmd/kube-proxy/app/server_others.go

@@ -571,7 +542,7 @@ func cleanupAndExit() error {

 	var encounteredError bool
 	for _, ipt := range ipts {
-		encounteredError = userspace.CleanupLeftovers(ipt) || encounteredError
+


possibly we should leave this for a while longer? (but if not, then you shouldn't leave a blank line behind)

cmd/kube-proxy/app/server.go

danwinship · 2022-08-31T14:46:58Z

cmd/kube-proxy/app/server_windows.go

-		}
-
-		winkernel.RegisterMetrics()
+	klog.V(0).InfoS("Using Kernelspace Proxier.")


not sure this is necessary to say any more since it's the only option?

cmd/kube-proxy/app/server_windows.go

danwinship · 2022-08-31T14:49:31Z

staging/src/k8s.io/kube-proxy/config/v1alpha1/types.go

@@ -196,7 +193,7 @@ type KubeProxyConfiguration struct {
 // In Linux platform, if proxy mode is blank, use the best-available proxy (currently iptables, but may change in the
 // future). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are
 // insufficient, this always falls back to the userspace proxy. IPVS mode will be enabled when proxy mode is set to 'ipvs',
-// and the fall back path is firstly iptables and then userspace.
+// and the fall back path is firstly iptables.
 //
 // In Windows platform, if proxy mode is blank, use the best-available proxy (currently userspace, but may change in the
 // future). If winkernel proxy is selected, regardless of how, but the Windows kernel can't support this mode of proxy,


looks like you failed to remove the docs text about windows userspace

test/e2e/framework/.import-restrictions

leilajal · 2022-09-01T16:39:38Z

/remove-sig api-machinery

jayunit100 · 2022-09-06T16:45:36Z

@daschott just fyi for headsup userspace is going awayyyy

jayunit100 · 2022-09-06T16:46:40Z

@ibabou FYI

daschott · 2022-09-06T17:06:27Z

cc @sbangari FYI

Enable windows proxyall feature by default because the kube-proxy userspace datapath has been removed since kubernetes 1.26. (kubernetes/kubernetes#112133) Signed-off-by: Shuyang Xin <gavinx@vmware.com>

knabben changed the title ~~Remove kube-proxy userspace modes~~ [wip] Remove kube-proxy userspace modes Aug 30, 2022

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework labels Aug 30, 2022

k8s-ci-robot requested a review from aojea August 30, 2022 21:03

k8s-ci-robot added the area/test label Aug 30, 2022

k8s-ci-robot requested a review from danwinship August 30, 2022 21:03

k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Aug 30, 2022

knabben force-pushed the remove-kproxy-userspace branch from cb9e949 to e399c34 Compare August 30, 2022 21:19

k8s-ci-robot added area/code-generation sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels Aug 30, 2022

knabben force-pushed the remove-kproxy-userspace branch from e399c34 to 92467bd Compare August 30, 2022 21:38

danwinship reviewed Aug 31, 2022

View reviewed changes

k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Sep 1, 2022

knabben force-pushed the remove-kproxy-userspace branch from 92467bd to 5f6db85 Compare September 7, 2022 02:08

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. labels Sep 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove kube-proxy userspace modes #112133

Remove kube-proxy userspace modes #112133

knabben commented Aug 30, 2022 •

edited by thockin

danwinship commented Aug 31, 2022

danwinship Aug 31, 2022

danwinship Aug 31, 2022

knabben Sep 7, 2022

danwinship Aug 31, 2022

danwinship Aug 31, 2022

knabben Aug 31, 2022

danwinship Aug 31, 2022

danwinship Aug 31, 2022

danwinship Aug 31, 2022

leilajal commented Sep 1, 2022

jayunit100 commented Sep 6, 2022

jayunit100 commented Sep 6, 2022

daschott commented Sep 6, 2022

Remove kube-proxy userspace modes #112133

Remove kube-proxy userspace modes #112133

Conversation

knabben commented Aug 30, 2022 • edited by thockin

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

danwinship commented Aug 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leilajal commented Sep 1, 2022

jayunit100 commented Sep 6, 2022

jayunit100 commented Sep 6, 2022

daschott commented Sep 6, 2022

knabben commented Aug 30, 2022 •

edited by thockin