Skip to content

runtime: limit number of operating system threads #4056

New issue
New issue
Closed
Closed
runtime: limit number of operating system threads#4056
Labels
FrozenDueToAge
Milestone
 
Go1.2
@rsc

Description

@rsc
rsc
opened on Sep 9, 2012
It comes up repeatedly that programs with large numbers of goroutines spawn far more
operating system threads than they can reasonably use, because the Go runtime is trying
not to find itself in a situation where every thread is blocked in the operating system.
Unfortunately this means that if something in the OS gets backed up Go just keeps making
more threads as more goroutines get stuck there. For example if DNS queries (done via
cgo) get stuck for a little while, then a program with 5000 HTTP-fetching goroutines
will end up with 5000 threads attempting DNS queries. The usual "solution" is
for callers of cgo code to limit by hand the number of goroutines entering that code.

I wonder if the runtime should expose a setting giving the maximum number of OS threads
to use. 256 or 512 could be a reasonable default - those seem quite high and all the
trouble I have seen has been with far more threads. It would probably be like 

// MaxOSThreads sets the maximum number of OS threads that will
// be used to run the current Go program and returns the previous setting.
// If n < 1, it does not change the current setting. The default is 256.
func MaxThreads(n int) int

Activity

bradfitz

bradfitz commented on Sep 10, 2012

@bradfitz
bradfitz
on Sep 10, 2012
Contributor

Comment 1:

Seems like you'd want to cap thread creation depending on the use.
If I have 256 blocked cgo DNS threads, that doesn't mean that a new thread to Read from
disk wouldn't make progress.
I understand that's more complex, though.  So I guess it could always be done by hand,
as it is now.
rsc

rsc commented on Sep 10, 2012

@rsc
rsc
on Sep 10, 2012
ContributorAuthor

Comment 2:

Yes, you can always raise or remove the limit and do things by hand. I
just think the default should be limited, not unlimited.
Russ
ianlancetaylor

ianlancetaylor commented on Sep 10, 2012

@ianlancetaylor
ianlancetaylor
on Sep 10, 2012
Contributor

Comment 3:

What if GOMAXPROCS > MaxOSThreads?
Do we want to limit the number of threads created to run cgo/SWIG, or do we want to
limit the total number of threads including those created to run goroutines that make
blocking system calls?
An absolute limit on all threads is easy to implement and understand but I'm concerned
that 256 is too low.
dvyukov

dvyukov commented on Sep 10, 2012

@dvyukov
dvyukov
on Sep 10, 2012
Member

Comment 4:

Thread pool with bounded number of threads known to cause problems. The first problem is
system-induced deadlocks. And it's not something that you will uncover during
unit-testing. Then, it may cause poor performance, e.g. I may have X threads almost
permanently blocked and I do not want them to affect the rest of the program. Then, it's
difficult for a user to set it correctly; so users may change the default when they do
not need to, or don't change when need to.
The new runtime partially solves the problem by "ignoring" short syscalls. On network
benchmarks with GOMAXPROCS=16 I see only about 25 threads. It may be further tuned by
increasing delay before cpu retake if there are already a lot of threads. If we are
committing the new runtime, then I would prefer to see whether it makes things better or
not before introducing user control.
Do we have a good reproducer for the problem?
rsc

rsc commented on Sep 10, 2012

@rsc
rsc
on Sep 10, 2012
ContributorAuthor

Comment 5:

Yes, I am aware that bounded thread pools cause problems. However,
unbounded ones cause problems too. At the moment I would be happy to
trade the bounded problems for the unbounded problems.
dvyukov

dvyukov commented on Sep 10, 2012

@dvyukov
dvyukov
on Sep 10, 2012
Member

Comment 6:

But that's not backwards-compatible change.
What about having unbounded number by default with an option to set the bound if
required?
ianlancetaylor

ianlancetaylor commented on Sep 10, 2012

@ianlancetaylor
ianlancetaylor
on Sep 10, 2012
Contributor

Comment 7:

I'm not worried about this sort of backward compatibility.
rsc

rsc commented on Sep 10, 2012

@rsc
rsc
on Sep 10, 2012
ContributorAuthor

Comment 8:

We're allowed to break compatibility for bug fixes. It is a bug that
if you have 10000 goroutines doing HTTP requests, a hiccup in your DNS
server triggers 10000 OS threads.
Russ
dvyukov

dvyukov commented on Sep 10, 2012

@dvyukov
dvyukov
on Sep 10, 2012
Member

Comment 9:

But it will break other programs that are not affected by the "bug".
rsc

rsc commented on Sep 11, 2012

@rsc
rsc
on Sep 11, 2012
ContributorAuthor

Comment 10:

That's why we have to provide a workaround API. If we set the default well
we will fix more programs than we break.
rsc

rsc commented on Sep 12, 2012

@rsc
rsc
on Sep 12, 2012
ContributorAuthor

Comment 12:

Labels changed: added go1.1.

bradfitz

bradfitz commented on Nov 8, 2012

@bradfitz
bradfitz
on Nov 8, 2012
Contributor

Comment 13:

Related: discussion on http://golang.org/cl/6815049 about number of threads
blocked in cgo DNS lookups.  Cap that too in the net package?
rsc

rsc commented on Dec 10, 2012

@rsc
rsc
on Dec 10, 2012
ContributorAuthor

Comment 14:

Labels changed: added size-m.

rsc

rsc commented on Dec 10, 2012

@rsc
rsc
on Dec 10, 2012
ContributorAuthor

Comment 15:

Labels changed: added suggested.

gopherbot

gopherbot commented on Feb 4, 2013

@gopherbot
gopherbot
on Feb 4, 2013
Contributor

Comment 16 by rickarnoldjr:

Attempted fix: https://golang.org/cl/7275049/

34 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAge

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @bradfitz@snaury@rsc@dvyukov@nightlyone

        Issue actions
          runtime: limit number of operating system threads · Issue #4056 · golang/go