Description
We should see if we can make ipvs do everything we need - it should perform even better than iptables. A benchmark is in order.
Notes:
root@kubernetes-minion-32zi:/home/thockin# ipvsadm -A -t 10.9.8.7:12345 -s rr
root@kubernetes-minion-32zi:/home/thockin# ipvsadm -a -t 10.9.8.7:12345 -m -r 10.244.1.27:9376
root@kubernetes-minion-32zi:/home/thockin# ipvsadm -a -t 10.9.8.7:12345 -m -r 10.244.1.28:9376
root@kubernetes-minion-32zi:/home/thockin# ip addr add 10.9.8.7/32 dev eth0
root@kubernetes-minion-32zi:/home/thockin# curl 10.9.8.7:12345
hostB
root@kubernetes-minion-32zi:/home/thockin# curl 10.9.8.7:12345
hostA
root@kubernetes-minion-32zi:/home/thockin# docker run -ti busybox wget -qO- 10.9.8.7:12345
hostB
root@kubernetes-minion-32zi:/home/thockin# docker run -ti busybox wget -qO- 10.9.8.7:12345
hostA
"masq" mode is DNAT not SNAT src ip is preserved.
We have to assign the VIP to some interface in the root NS. This is a bit ugly in that ports NOT exposed by the VIP get sent to the host (e.g. 22). I think we can fix that by adding another catchall for the VIP. I don't know if there are limits to local IPs
Not sure if there is a atomic batch update command, but it does handle batch invocation at least.
Several scheduling policies, but rr
seems sufficient, maybe lc
. sh
seems to give us client affinity.
We can configure timeouts.
We'll need to do something for node-ports, probably still iptables. I think this (and the other tricks we pull for load-balancers) will be the biggest challenge.
@BenTheElder busy? :)
Activity
BenTheElder commentedon Nov 19, 2015
Pretty busy right now, last round of midterms and final projects right now then soon-ish I have finals. We're out for the holidays in about 3 weeks though (done for sure by December 12th).
I'll be sure to take a look if/when I can find the time though!
thockin commentedon Nov 19, 2015
I was kidding :)
On Nov 18, 2015 5:42 PM, "Benjamin Elder" notifications@github.com wrote:
BenTheElder commentedon Nov 19, 2015
Ah, whizzed right over my head. :)
I very much enjoy working in OSS though, If I don't get wrapped up in
something else I may have to get back into k8s tinkering again.
I'll stop cluttering this issue for now though :)
On Wed, Nov 18, 2015 at 9:55 PM, Tim Hockin notifications@github.com
wrote:
hw-qiaolei commentedon Nov 23, 2015
@thockin I like this idea. Use
iptables
for LB seems limited(for LB algorithms) and less graceful(thousands ofiptables
rules).I noticed Andrey Sibiryov who is from Uber had also given a session "Kernel load-balancing for Docker containers using IPVS" on DockerCon 2015 eu. Please see DockerCon 2015 eu Agenda.
thockin commentedon Nov 23, 2015
Yeah, I think this is actually not a very hard project, but I'd want to see
some graphs.
On Sun, Nov 22, 2015 at 11:04 PM, qiaolei notifications@github.com wrote:
aledbf commentedon Dec 4, 2015
Video DockerCon 2015 eu Kernel load-balancing for Docker containers using IPVS
thockin commentedon Dec 4, 2015
yeah, IPVS works. I tried it out a few months back, but I was missing a
piece of the recipe.
On Thu, Dec 3, 2015 at 8:09 PM, Manuel Alejandro de Brito Fontes <
notifications@github.com> wrote:
guybrush commentedon Dec 4, 2015
also it would be cool then to have k8s-services utilize the ipvs features, like persistence and selecting the balance-strategy (and even weights?)
guilhem commentedon Dec 8, 2015
Remind me something :) #3760 (comment)
feiskyer commentedon Jan 11, 2016
Interesting.
qoke commentedon Jan 12, 2016
Whilst poking around for other threads, I found this... moby/libnetwork#852
and this https://github.com/kobolog/gorb which may be of interest..
50 remaining items