Skip to content

Try kube-proxy via ipvs instead of iptables or userspace #17470

Closed
@thockin

Description

@thockin
Member

We should see if we can make ipvs do everything we need - it should perform even better than iptables. A benchmark is in order.

Notes:

root@kubernetes-minion-32zi:/home/thockin# ipvsadm -A -t 10.9.8.7:12345 -s rr
root@kubernetes-minion-32zi:/home/thockin# ipvsadm -a -t 10.9.8.7:12345 -m -r 10.244.1.27:9376
root@kubernetes-minion-32zi:/home/thockin# ipvsadm -a -t 10.9.8.7:12345 -m -r 10.244.1.28:9376

root@kubernetes-minion-32zi:/home/thockin# ip addr add 10.9.8.7/32 dev eth0

root@kubernetes-minion-32zi:/home/thockin# curl 10.9.8.7:12345
hostB

root@kubernetes-minion-32zi:/home/thockin# curl 10.9.8.7:12345
hostA

root@kubernetes-minion-32zi:/home/thockin# docker run -ti busybox wget -qO- 10.9.8.7:12345
hostB

root@kubernetes-minion-32zi:/home/thockin# docker run -ti busybox wget -qO- 10.9.8.7:12345
hostA

"masq" mode is DNAT not SNAT src ip is preserved.

We have to assign the VIP to some interface in the root NS. This is a bit ugly in that ports NOT exposed by the VIP get sent to the host (e.g. 22). I think we can fix that by adding another catchall for the VIP. I don't know if there are limits to local IPs

Not sure if there is a atomic batch update command, but it does handle batch invocation at least.

Several scheduling policies, but rr seems sufficient, maybe lc. sh seems to give us client affinity.

We can configure timeouts.

We'll need to do something for node-ports, probably still iptables. I think this (and the other tricks we pull for load-balancers) will be the biggest challenge.

@BenTheElder busy? :)

Activity

BenTheElder

BenTheElder commented on Nov 19, 2015

@BenTheElder
Member

Pretty busy right now, last round of midterms and final projects right now then soon-ish I have finals. We're out for the holidays in about 3 weeks though (done for sure by December 12th).

I'll be sure to take a look if/when I can find the time though!

thockin

thockin commented on Nov 19, 2015

@thockin
MemberAuthor

I was kidding :)
On Nov 18, 2015 5:42 PM, "Benjamin Elder" notifications@github.com wrote:

Pretty busy right now, last round of midterms and final projects right now
then soon-ish I have finals. We're out for the holidays in about 3 weeks
though (December 12th).

I'll be sure to take a look if/when I can find the time though!


Reply to this email directly or view it on GitHub
#17470 (comment)
.

BenTheElder

BenTheElder commented on Nov 19, 2015

@BenTheElder
Member

Ah, whizzed right over my head. :)

I very much enjoy working in OSS though, If I don't get wrapped up in
something else I may have to get back into k8s tinkering again.

I'll stop cluttering this issue for now though :)

On Wed, Nov 18, 2015 at 9:55 PM, Tim Hockin notifications@github.com
wrote:

I was kidding :)
On Nov 18, 2015 5:42 PM, "Benjamin Elder" notifications@github.com
wrote:

Pretty busy right now, last round of midterms and final projects right
now
then soon-ish I have finals. We're out for the holidays in about 3 weeks
though (December 12th).

I'll be sure to take a look if/when I can find the time though!


Reply to this email directly or view it on GitHub
<
#17470 (comment)

.


Reply to this email directly or view it on GitHub
#17470 (comment)
.

hw-qiaolei

hw-qiaolei commented on Nov 23, 2015

@hw-qiaolei
Contributor

@thockin I like this idea. Use iptables for LB seems limited(for LB algorithms) and less graceful(thousands of iptables rules).

I noticed Andrey Sibiryov who is from Uber had also given a session "Kernel load-balancing for Docker containers using IPVS" on DockerCon 2015 eu. Please see DockerCon 2015 eu Agenda.

thockin

thockin commented on Nov 23, 2015

@thockin
MemberAuthor

Yeah, I think this is actually not a very hard project, but I'd want to see
some graphs.

On Sun, Nov 22, 2015 at 11:04 PM, qiaolei notifications@github.com wrote:

@thockin https://github.com/thockin I like this idea. Use iptables for
LB seems limited(for LB algorithms) and less graceful(thousands of
iptables rules).

I noticed Andrey Sibiryov who is from Uber had also given a session
"Kernel load-balancing for Docker containers using IPVS" on DockerCon 2015
eu. Please see DockerCon 2015 eu Agenda
http://europe-2015.dockercon.com/agenda.


Reply to this email directly or view it on GitHub
#17470 (comment)
.

aledbf

aledbf commented on Dec 4, 2015

@aledbf
Member
thockin

thockin commented on Dec 4, 2015

@thockin
MemberAuthor

yeah, IPVS works. I tried it out a few months back, but I was missing a
piece of the recipe.

On Thu, Dec 3, 2015 at 8:09 PM, Manuel Alejandro de Brito Fontes <
notifications@github.com> wrote:

Video DockerCon 2015 eu Kernel load-balancing for Docker containers using
IPVS https://www.youtube.com/watch?v=oFsJVV1btDU


Reply to this email directly or view it on GitHub
#17470 (comment)
.

guybrush

guybrush commented on Dec 4, 2015

@guybrush

also it would be cool then to have k8s-services utilize the ipvs features, like persistence and selecting the balance-strategy (and even weights?)

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector: 
    project: my-service
  ports:
    - protocol: "TCP"
      port: 80
      targetPort: 80
      strategy: "rr"
      persistence: true
guilhem

guilhem commented on Dec 8, 2015

@guilhem

Remind me something :) #3760 (comment)

feiskyer

feiskyer commented on Jan 11, 2016

@feiskyer
Member

Interesting.

qoke

qoke commented on Jan 12, 2016

@qoke

Whilst poking around for other threads, I found this... moby/libnetwork#852
and this https://github.com/kobolog/gorb which may be of interest..

50 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/kube-proxysig/networkCategorizes an issue or PR as relevant to SIG Network.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @matthiasr@aledbf@kevinz@guybrush@kobolog

      Issue actions

        Try kube-proxy via ipvs instead of iptables or userspace · Issue #17470 · kubernetes/kubernetes