Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Docker should assist in bandwidth limiting containers #26767

Open
vincentwoo opened this issue Sep 21, 2016 · 46 comments
Open

Proposal: Docker should assist in bandwidth limiting containers #26767

vincentwoo opened this issue Sep 21, 2016 · 46 comments
Labels
area/networking exp/expert kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny

Comments

@vincentwoo
Copy link
Contributor

vincentwoo commented Sep 21, 2016

In one of the very earliest Docker issues, #37: Add ability to throttle bandwidth by container, Ken proposes that Docker should allow per-container bandwidth throttling support. I think this idea should be reconsidered.

The current state of affairs is that people who want to do this are told to use tc on the veth for a given container. I believe this to not great place to be for a few reasons:

  1. tc is notoriously difficult to learn, so much so that various different suites of software have tried to paper over its rough edges. All of them are unsuited for use with docker and the large number of veths it creates.
  2. tc has to be configured at container creation time. If you are deploying at scale, containers will be brought up or down constantly, which will force you, a developer or operations person, to constantly bookkeep to keep tc up to date. In practice this means you will give up. This problem is especially bad if you create many short lived containers per minute (ahem, me).
  3. As the Docker-centric ecosystem expands, more and more critical ops behavior will need to apply to specific containers. Right now, cluster management is focused on the allocation and distribution of containers across physical nodes, resource allotment, and load balancing / discovery. Rate limiting and more fine-grained resource allotment (ahem, disk quota) are logical next steps, and the network throttling side of all this has been neglected.

I propose that Docker manage underlying network rules (however they are represented on the underlying system), to accept (at minimum) maximum bandwidth limits for inbound and outbound traffic, per-container. This should probably be implemented as an attribute on container creation.

THOUGHTS?

@icecrime icecrime added exp/expert kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny area/networking labels Sep 21, 2016
@xiaods
Copy link
Contributor

xiaods commented Sep 21, 2016

@vincentwoo i know libnetwork will handle overall network implement, so how about move the proposal to libnetwork issues?

@vincentwoo
Copy link
Contributor Author

I'm not against the idea, but I have no idea if that's the right place or not. Can anyone else chime in? For instance, since this proposal would (should?) require changing the API surface of container creation, I'm not entirely sure it 100% belongs in libnetwork. More broadly, I think people need to decide if this is even something they want Docker to do.

@justincormack
Copy link
Contributor

justincormack commented Sep 21, 2016

Also the question is whether you want to limit bandwidth or prioritise. The net_cls cgroup can classify traffic for tc, but not sure where policy should be determined. It will depend on your network driver to some extent.

It would definitely help for people to detail their real world use cases for this.

@vincentwoo
Copy link
Contributor Author

vincentwoo commented Sep 21, 2016

Great point. My real world use case is that I run many ephemeral containers, ranging in lifespan from a second to an hour, on behalf of many users, some of whom are potentially malicious.

Many of the common ways to abuse the system have been addressed, but being able to strictly limit bandwidth (not prioritize) per-container using individual quotas is one of the last remaining issues.

@thaJeztah
Copy link
Member

thaJeztah commented Sep 21, 2016

Also related; #4763, #9607, #16477

@justincormack
Copy link
Contributor

@vincentwoo are those per container limits or per some grouping of containers? (eg all the containers belonging to user X can have 1Mb/s)?

@vincentwoo
Copy link
Contributor Author

Per-container

@vincentwoo
Copy link
Contributor Author

vincentwoo commented Sep 21, 2016

See also #17064, "[RFE] Allow docker to report the veth interface used by a container" - stated use case on that proposal is ALSO to achieve use of tc.

@aboch
Copy link
Contributor

aboch commented Sep 21, 2016

From implementation point of view I think this can be done by libnetwork itself when it programs the container's interface, so without drivers involvement.

But agree the API should be sorted out first.

For example, maybe it makes also sense a bandwidth policy per network, where one may want to create one unlimited and one limited network and place containers on each network accordingly ?

@vincentwoo
Copy link
Contributor Author

I think largely people will want to configure:

  1. Groups of containers with roughly equal quota
  2. Individual containers with specific quotas

against various throttling strategies:

  1. Max rate with bursting
  2. Max flat rate
  3. Max total data transferred in time window

@aboch aboch self-assigned this Sep 26, 2016
@vincentwoo
Copy link
Contributor Author

Hi @aboch, I saw that you self-assigned this and #21990. Can you give us a sense for where your thinking's at on this topic? Do we need to build additional consensus, etc? I'm unsure how the proposal process works in practice.

@aboch
Copy link
Contributor

aboch commented Oct 4, 2016

@vincentwoo
Given both requested features require libnetwork changes, I self-assigned so that I can monitor the discussions/consensus, see if/how the requested funcionality fits in libnetwork and answer questions that may arise about current implementation or feasibility.

To be clear, this is not on the roadmap yet.

As @justincormack suggested, it is important to hear use-cases where this will be used (thanks for providing yours), ideally from more than one party so that we can can come out with an hopefully good abstraction to accomodate the most common cases.

Besides that, community is at any time free/welcome to push a PR with a proposed implemenation to address this. That can also speed up the process.

@termie
Copy link
Contributor

termie commented Oct 4, 2016

+1, we would use this in a second at wercker and I agree exactly with vincent's 3 reasons for it not being viable to manage this outside of docker

@lhmerino
Copy link

lhmerino commented Oct 5, 2016

+1, I agree as well. It would be great especially if a container would to be compromised and a hacker launched an DOS attack from the container to an outside party.

@vivekjuneja
Copy link

+1 for this as well. We are planning to use Filebeat as a Docker Container. We use NIC bonding to provide a single network interface for application and logs. Isolating one NIC from the bond is something that we cannot control, as we use on-demand VMs (standardized). So, isolating the Filebeat traffic from App Services would be a great use of this feature.

@salmanbaset
Copy link

+1 - this should be a docker run option at least for bridged networking. Initially, something like:
docker run --net-out-bps | --net-in-bps
with options for bursting possibly added later.

underneath, it can configure tc and tear it down.

@shivacherukuri
Copy link

Hi,
I have done implementation of network bandwidth management, please look at the the following link for more details and provide me feedback.
https://github.com/shivacherukuri/Docker-Network-Bandwidth

@vincentwoo
Copy link
Contributor Author

I'm uh, a bit confused in following your explanations, but it seems to me that this idea will only work for network bandwidth into a container on a very predefined port.

@shivacherukuri
Copy link

prototype done for predefined port and also for a full container. I mean if we start a container by exposing a public host port then this new feature start limiting egress and ingress n/w io based on the port and if we just start a container without specifiying/exposing a port then it will limit based on the container net_cls cgroup class id

@Rid
Copy link
Contributor

Rid commented Nov 14, 2016

@shivacherukuri how did you get around the issue in #19802? I'm running Ubuntu 16.04 with 4.4.0-47-generic and net_cls is not being marked on packets coming from the container, although it works fine on any process outside the container.

@spkane
Copy link
Contributor

spkane commented Apr 18, 2017

This ticket has been quiet for awhile, even after at least some work on a potential implementation approach. Is there any discussion happening behind the scenes regarding bandwidth controls on containers and how this might actually be implemented.

In my case, I'm mostly looking for a way to throttle individual containers, with the realization that outbound is likely easier than inbound, to control in a meaning full way.

@spkane
Copy link
Contributor

spkane commented Apr 18, 2017

@aboch I have no idea how you just got unassigned, but apparently I can't add you back.

@mmaquevice
Copy link

mmaquevice commented Aug 29, 2017

Any updates?

@thaJeztah
Copy link
Member

/cc @fcrisciani @abhinandanpb any thoughts?

@vincentwoo
Copy link
Contributor Author

OP here, would love to hear what next steps for this request might be. Does the tram need to see more user need?

@deweysasser
Copy link

My use-case: I regularly run backups using docker containers. I'd like to limit the (outbound) bandwidth of the container to avoid saturating my network pipe. As many tools are using http or worse, https for communication, putting tc rules at the router level based on protocol is not functional.

@vincentwoo
Copy link
Contributor Author

Hi, just checking back in. I still would love to know what next steps are relevant for this request.

@jan-g
Copy link

jan-g commented Feb 7, 2018

Also have a use-case to filter traffic (all traffic) originating in a container into a class on the host corresponding to that container; this for managing some rate-limiting but with borrowing between specific classes. Getting the veth would probably be enough (or having docker set up mark iptables entries for me) since I can hook into the lifecycle at the point of docker creation and take care of the rest.

@gregg-saffell
Copy link

I would like to see something like Docker's CPU throttling for bandwidth. If I understand correctly, CPU limits allows for a relative amount to be allocated and Docker will only enforce the limit when there is contention for the resource. That would be ideal because it would allow me to guarantee a percentage of the bandwidth in peak times while allowing the network to fully leveraged during non-peak times.

@jmwilkinson
Copy link

This would be an incredibly useful feature, especially for multi-tenant supported environments. The main need is bytes/time window.

Where are we at on this? Any update?

@cpuguy83
Copy link
Member

@jmwilkinson No one has taken this work on AFAIK.

@xiaods
Copy link
Contributor

xiaods commented Sep 10, 2018

any insight comment on it?

@vrosnet

This comment was marked as spam.

@vincentwoo
Copy link
Contributor Author

I'm interested in funding this work if anyone's looking to take this on. Feel free to email me (address in my profile).

@vrosnet

This comment was marked as spam.

@shivacherukuri
Copy link

Till Oct 15, 2018, I am damn committed. I will try talking to you after that.. seems like many are still interested in this feature. we may need to brainstorm on this topic again if we need to work on the implementation.

@shellus
Copy link

shellus commented Apr 12, 2019

Any updates?

1 similar comment
@timchenxiaoyu
Copy link
Contributor

Any updates?

@vincentwoo
Copy link
Contributor Author

No one's mailed me but I'm still interested in funding this work.

@xiaods
Copy link
Contributor

xiaods commented May 21, 2019

@vincentwoo can we implement it with plugin?

@Slitzo3
Copy link

Slitzo3 commented Mar 26, 2020

Any updates on this?

@Meganitrospeed
Copy link

Any updates on this ?

@Roki100
Copy link

Roki100 commented Mar 17, 2022

Any updates??? This is one of the long-requested things that are possibly not even considered yet

@ecmonsen
Copy link

+1 I would use this for testing and prototyping scenarios where I need to simulate a slow network connection.

@echobom
Copy link

echobom commented Nov 25, 2023

Any updates???

@AWonderfulUser
Copy link

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/networking exp/expert kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny
Projects
None yet
Development

No branches or pull requests