Implement IPVS-based in-cluster service load balancing #44063

ghost · 2017-04-04T23:37:07Z

At KubeCon Europe in Berlin last week I presented some work we've done at Huawei scaling Kubernetes in-cluster load balancing to 50,000+ services and beyond, the challenges associated with doing this using the current iptables approach, and what we've achieved using an alternative IPVS-based approach. iptables is designed for firewalling, and based on in-kernel rule lists, while IPVS is designed for load balancing and based on in-kernel hash tables. IPVS also supports more sophisticated load balancing algorithms than iptables (least load, least conns, locality, weighted) as well as other useful features (e.g. health checking, retries etc).

After the presentation, there was strong support (a.k.a. a riot :-) ) for us to open source this work, which we are happy to do. We can use this issue to track that.

For those who were not able to be there, here is the video:

https://youtu.be/c7d_kD2eH4w

And the slides:

https://docs.google.com/presentation/d/1BaIAywY2qqeHtyGZtlyAp89JIZs59MZLKcFLxKE6LyM/edit?usp=sharing

We will follow up on this with a more formal design proposal, and a set of PR's, but in summary we added a about 680 lines of code to the existing 12,000 lines of kube-proxy (~5%), and added a third mode flag to it's command-line (mode=IPVS, to the existing mode=userspace and mode=iptables).
Performance improvement of load balancer updates is dramatic (update latency reduced from hours per rule to 2ms per rule). Network latency and variability also reduced dramatically for large numbers of services.

@kubernetes/sig-network-feature-requests
@kubernetes/sig-scalability-feature-requests
@thockin
@wojtek-t

gyliu513 · 2017-04-05T03:31:06Z

Even though Kubernetes 1.6 support 5000 nodes, but the kube-proxy with iptables is actually a bottleneck to scale the cluster to 5000 nodes. One example is that with NodePort service in a 5000 node cluster, if I have 2000 services and each services have 10 pods, this will cause 20000 iptable records on each worker node, and this can make the kernel pretty busy. Using IPVS-based in-cluster service load balancing can help a lot for such case.

ravilr · 2017-04-05T06:10:34Z

@quinton-hoole is your implementation using IPVS in nat mode or Direct routing mode?

haibinxie · 2017-04-05T16:29:13Z

@ravilr it's NAT mode.

timothysc · 2017-04-05T18:14:41Z

gr8 work, the iptables issues has been a problem for a while.

re: flow based scheduling, happy to help get the firmament scheduler in place. ;-)
/cc @kubernetes/sig-scheduling-feature-requests

resouer · 2017-04-06T07:27:20Z

@timothysc I paid attention to firmament for a period of time, but not quite get it's value adding to kubernetes. Would you mind to explain what problem flow based scheduling can solve in current Kubernetes scheduler?

timothysc · 2017-04-06T14:50:53Z

@resouer speed at scale and rescheduling.

From @quinton-hoole 's talk, linked above, it looks like huawei has been prototyping this.

ghost · 2017-04-06T15:25:23Z

@resouer @timothysc Yes, I can confirm that we're working on a Firmament scheduler, and will upstream it as soon as it's in good enough shape. We might have an initial implementation in the next few weeks.

deepak-vij · 2017-04-06T15:48:18Z

Hi folks, we are currently working on implementing Firmament Scheduler as part of K8S scheduling environment. We will create a new separate issue to track the progress and provide updates etc. thanks.

sureshvis · 2017-04-06T16:02:06Z

@quinton-hoole thanks for sharing. Waiting to see the design proposal.

In term of Healthcheck, every worker doing health check across all pods to keep the table upto date?, how are you planning to handle this @scale ?

ghost · 2017-04-06T16:14:17Z

cc @haibinxie

ghost · 2017-04-06T16:20:37Z

To be clear, @haibinxie did all the hard work here. Please direct questions to him.

MikeSpreitzer · 2017-04-06T18:18:24Z

IPVS only deals with IP, not transport protocols, right? A k8s service can include a port transformation. A Service object has a potential distinction between port and targetPort in the items in the spec.ports list. And an Endpoints also has a ports.port field in items in the subsets list. Can your implementation handle this generality, and if not, what happens when the user asks for it?

haibinxie · 2017-04-06T20:14:13Z

@MikeSpreitzer Port transformation is well supported.

thockin · 2017-04-10T05:13:32Z

Note that even with IPVS, NodePorts, firewalls, and so on still have to be handled and the obvious place to handle much of it is iptables.

…

On Tue, Apr 4, 2017 at 8:31 PM, Guangya Liu ***@***.***> wrote: Even though Kubernetes 1.6 support 5000 nodes, but the kube-proxy with iptables is actually a bottleneck to scale the cluster to 5000 nodes. One example is that with NodePort service in a 5000 node cluster, if I have 2000 services and each services have 10 pods, this will cause 20000 iptable records on each worker node, and this can make the kernel pretty busy. Using IPVS-based in-cluster service load balancing can help a lot for such case. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44063 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVIWXR-AXPGcK7veFpIIhkdQLFspnks5rswsMgaJpZM4MzkIA> .

thockin · 2017-04-10T05:16:50Z

Port mapping is handled in NAT mode (called masquerade by IPVS, sadly). As an optimization, a future followup could enable direct-return mode for environments that support it for services that do not do remapping. We'd have to add service IPs as local addresses in pods, which we may want to do anyway.

thockin · 2017-04-10T05:21:37Z

Last comment for the record here, though I have said it elsewhere.

I am very much in favor of an IPVS implementation. We have somewhat more than JUST load-balancing in our iptables (session affinity, firewalls, hairpin-masquerade tricks), but I believe those can all be overcome.

We also have been asked, several times, to add support for port ranges to Services, up to and including a whole IP. The obvious way to add this would also support remapping, though it is not at all clear how NodePorts would work. IPVS, as far as I know, has no facility for exposing ranges of ports.

ChenLingPeng · 2017-04-10T07:07:58Z

In IPVS mode, we have to add all the service address to host device like lo or ethx, am I right?

haibinxie · 2017-05-09T20:40:31Z

Hi All, I put together a proposal for the alpha version of IPVS implementation hoping to get into kubernetes 1.7. need your feedback.

https://docs.google.com/document/d/1YEBWR4EWeCEWwxufXzRM0e82l_lYYzIXQiSayGaVQ8M/edit?usp=sharing

@kubernetes/sig-network-feature-requests
@kubernetes/sig-scalability-feature-requests
@thockin
@wojtek-t

dhilipkumars · 2017-05-22T12:53:25Z

FYI
if docker accepts this PR, we may be able to loose seesaw (libnl.so) dependency. moby/libnetwork#1770

gyliu513 · 2017-05-25T05:36:50Z

Does the kube-router can help this?

thockin · 2017-05-27T06:19:51Z

Have we seen code for this yet? At this point, there's simply no way it is making v1.7, but I'd love to queue up something here for v1.8

…

On Wed, May 24, 2017 at 10:37 PM, Guang Ya Liu ***@***.***> wrote: Does the kube-router <https://github.com/cloudnativelabs/kube-router> can help this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44063 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVLs2Jmug8fsBHLEHsj_CyKZW9tXWks5r9ROCgaJpZM4MzkIA> .

ghost · 2017-05-27T21:11:26Z

Code is done and going through final e2e testing. The plan is to get it into v1.8.

…

On May 26, 2017 23:20, "Tim Hockin" ***@***.***> wrote: Have we seen code for this yet? At this point, there's simply no way it is making v1.7, but I'd love to queue up something here for v1.8 On Wed, May 24, 2017 at 10:37 PM, Guang Ya Liu ***@***.***> wrote: > Does the kube-router <https://github.com/cloudnativelabs/kube-router> can > help this? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/kubernetes/kubernetes/issues/ 44063#issuecomment-303927263>, > or mute the thread > <https://github.com/notifications/unsubscribe- auth/AFVgVLs2Jmug8fsBHLEHsj_CyKZW9tXWks5r9ROCgaJpZM4MzkIA> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44063 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ6NAdE8hjCgsuLh35NAIesHyvhWtiPVks5r98CkgaJpZM4MzkIA> .

thockin · 2017-05-27T22:41:41Z

But nobody has seen it yet? Publish early and often. You *know* there's going to be a lot of feedback - get that code out now now now. Don't sit on it. Don't polish it. PUBLISH. Or risk that it doesn't get merged because of the depth of feedback, or that someone else implements it while you're busy polishing. We have a fairly robust e2e suite. You should be able to just boot with this enabled and if it passes e2e, that's a strong signal. On Sat, May 27, 2017 at 2:12 PM, Quinton Hoole <notifications@github.com> wrote:

…

Code is done and going through final e2e testing. The plan is to get it into v1.8. On May 26, 2017 23:20, "Tim Hockin" ***@***.***> wrote: > Have we seen code for this yet? At this point, there's simply no way it is > making v1.7, but I'd love to queue up something here for v1.8 > > On Wed, May 24, 2017 at 10:37 PM, Guang Ya Liu ***@***.*** > > wrote: > > > Does the kube-router <https://github.com/cloudnativelabs/kube-router> > can > > help this? > > > > — > > You are receiving this because you were mentioned. > > Reply to this email directly, view it on GitHub > > <https://github.com/kubernetes/kubernetes/issues/ > 44063#issuecomment-303927263>, > > or mute the thread > > <https://github.com/notifications/unsubscribe- > auth/AFVgVLs2Jmug8fsBHLEHsj_CyKZW9tXWks5r9ROCgaJpZM4MzkIA> > > . > > > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/kubernetes/kubernetes/issues/ 44063#issuecomment-304431416>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ AJ6NAdE8hjCgsuLh35NAIesHyvhWtiPVks5r98CkgaJpZM4MzkIA> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44063 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVPp-EpTDj4x7dNZAi2LcgklBy9Xqks5r-JGggaJpZM4MzkIA> .

dujun1990 · 2017-05-28T11:54:04Z

@thockin @quinton-hoole

Initial PR #46580 is already sent out. PTAL.

dhilipkumars · 2017-05-28T13:26:41Z

@thockin originally we were relying on seesaw library and had a plan of updating it to a pure go implementation as phase 2 (probably in 1.8). Because of the complexities introduced by libnl.so dependencies last week we decided to move away from seesaw. Docker's libnetwork had a good set of ipvs apis but was missing GETXXX() methods. We quicky contributed to libnetwork and that got merged 3 days ago. Now we have vendored libnetwork. PTAL.

@thockin

Automatic merge from submit-queue (batch tested with PRs 51377, 46580, 50998, 51466, 49749) Implement IPVS-based in-cluster service load balancing **What this PR does / why we need it**: Implement IPVS-based in-cluster service load balancing. It can provide some performance enhancement and some other benefits to kube-proxy while comparing iptables and userspace mode. Besides, it also support more sophisticated load balancing algorithms than iptables (least conns, weighted, hash and so on). **Which issue this PR fixes** #17470 #44063 **Special notes for your reviewer**: * Since the PR is a bit large, I splitted it and move the commits related to ipvs util pkg to PR #48994. Hopefully can make it easier to review. @thockin @quinton-hoole @kevin-wangzefeng @deepak-vij @haibinxie @dhilipkumars @fisherxu **Release note**: ```release-note Implement IPVS-based in-cluster service load balancing ```

m1093782566 · 2017-11-20T06:07:35Z

/area ipvs

m1093782566 · 2017-12-07T03:44:08Z

IPVS-based kube-proxy is in beta phase now.

fejta-bot · 2018-03-07T04:18:16Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2018-04-15T19:20:53Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot · 2018-05-15T20:08:52Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

chrishiestand · 2018-05-30T18:34:57Z

With this issue closed as stale, is there a better issue to follow progress of adding ipvs scheduling algorithms to individual kubernetes services? I couldn't find another issue that explicitly covers this part of the ipvs roadmap.

arzarif · 2019-02-16T04:40:44Z

@chrishiestand

It's mentioned here as a future possibility, but like you, I haven't been able to find an issue where this is being actively discussed/worked on. Does anyone know whether there's ongoing work to make service-specific load-balancing algorithms possible?

ghost added area/kube-proxy kind/feature Categorizes issue or PR as related to a new feature. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. team/huawei labels Apr 4, 2017

ghost mentioned this issue Apr 6, 2017

Test IPVS kube-proxy mozmeao/infra#129

Closed

szibis mentioned this issue May 12, 2017

integrate with kubernetes kobolog/gorb#32

Closed

thockin added sig/network Categorizes an issue or PR as relevant to SIG Network. and removed sig/network Categorizes an issue or PR as relevant to SIG Network. labels May 16, 2017

dujun1990 mentioned this issue May 23, 2017

[e2e test] Why should check kube-proxy urls case always assume proxy mode is iptables? #46078

Closed

dujun1990 mentioned this issue May 28, 2017

Implement IPVS-based in-cluster service load balancing #46580

Merged

m1093782566 mentioned this issue Jun 23, 2017

support ipvs mode for kube-proxy kubernetes/community#692

Closed

emmanuel mentioned this issue Jun 23, 2017

Elasticsearch evaluation for MDN mozmeao/infra#193

Closed

k8s-ci-robot added the area/ipvs label Nov 20, 2017

vincentmli mentioned this issue Feb 28, 2018

Kube-flannel pod Failed to create SubnetManager: error retrieving pod spec dial tcp 10.0.0.1:443: i/o timeout #60161

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 7, 2018

spiffxp removed the team/huawei label Mar 16, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 15, 2018

k8s-ci-robot closed this as completed May 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement IPVS-based in-cluster service load balancing #44063

Implement IPVS-based in-cluster service load balancing #44063

ghost commented Apr 4, 2017 •

edited by ghost

gyliu513 commented Apr 5, 2017

ravilr commented Apr 5, 2017

haibinxie commented Apr 5, 2017

timothysc commented Apr 5, 2017

resouer commented Apr 6, 2017

timothysc commented Apr 6, 2017

ghost commented Apr 6, 2017

deepak-vij commented Apr 6, 2017

sureshvis commented Apr 6, 2017

ghost commented Apr 6, 2017

ghost commented Apr 6, 2017

MikeSpreitzer commented Apr 6, 2017

haibinxie commented Apr 6, 2017

thockin commented Apr 10, 2017 via email

thockin commented Apr 10, 2017

thockin commented Apr 10, 2017

ChenLingPeng commented Apr 10, 2017

haibinxie commented May 9, 2017

dhilipkumars commented May 22, 2017

gyliu513 commented May 25, 2017

thockin commented May 27, 2017 via email

ghost commented May 27, 2017 via email

thockin commented May 27, 2017 via email

dujun1990 commented May 28, 2017 •

edited

dhilipkumars commented May 28, 2017

m1093782566 commented Nov 20, 2017

m1093782566 commented Dec 7, 2017

fejta-bot commented Mar 7, 2018

fejta-bot commented Apr 15, 2018

fejta-bot commented May 15, 2018

chrishiestand commented May 30, 2018

arzarif commented Feb 16, 2019

Implement IPVS-based in-cluster service load balancing #44063

Implement IPVS-based in-cluster service load balancing #44063

Comments

ghost commented Apr 4, 2017 • edited by ghost

gyliu513 commented Apr 5, 2017

ravilr commented Apr 5, 2017

haibinxie commented Apr 5, 2017

timothysc commented Apr 5, 2017

resouer commented Apr 6, 2017

timothysc commented Apr 6, 2017

ghost commented Apr 6, 2017

deepak-vij commented Apr 6, 2017

sureshvis commented Apr 6, 2017

ghost commented Apr 6, 2017

ghost commented Apr 6, 2017

MikeSpreitzer commented Apr 6, 2017

haibinxie commented Apr 6, 2017

thockin commented Apr 10, 2017 via email

thockin commented Apr 10, 2017

thockin commented Apr 10, 2017

ChenLingPeng commented Apr 10, 2017

haibinxie commented May 9, 2017

dhilipkumars commented May 22, 2017

gyliu513 commented May 25, 2017

thockin commented May 27, 2017 via email

ghost commented May 27, 2017 via email

thockin commented May 27, 2017 via email

dujun1990 commented May 28, 2017 • edited

dhilipkumars commented May 28, 2017

m1093782566 commented Nov 20, 2017

m1093782566 commented Dec 7, 2017

fejta-bot commented Mar 7, 2018

fejta-bot commented Apr 15, 2018

fejta-bot commented May 15, 2018

chrishiestand commented May 30, 2018

arzarif commented Feb 16, 2019

ghost commented Apr 4, 2017 •

edited by ghost

dujun1990 commented May 28, 2017 •

edited