Description
I propose that we solve #10958 (preemption of tight loops) using non-cooperative preemption techniques. I have a detailed design proposal, which I will post shortly. This issue will track this specific implementation approach, as opposed to the general problem.
Edit: Design doc
Currently, Go currently uses compiler-inserted cooperative preemption points in function prologues. The majority of the time, this is good enough to allow Go developers to ignore preemption and focus on writing clear parallel code, but it has sharp edges that we've seen degrade the developer experience time and time again. When it goes wrong, it goes spectacularly wrong, leading to mysterious system-wide latency issues (#17831, #19241) and sometimes complete freezes (#543, #12553, #13546, #14561, #15442, #17174, #20793, #21053). And because this is a language implementation issue that exists outside of Go's language semantics, these failures are surprising and very difficult to debug.
@dr2chase has put significant effort into prototyping cooperative preemption points in loops, which is one way to solve this problem. However, even sophisticated approaches to this led to unacceptable slow-downs in tight loops (where slow-downs are generally least acceptable).
I propose that the Go implementation switch to non-cooperative preemption using stack and register maps at (essentially) every instruction. This would allow goroutines to be preempted without explicit
preemption checks. This approach will solve the problem of delayed preemption with zero run-time overhead and have side benefits for debugger function calls (#21678).
I've already prototyped significant components of this solution, including constructing register maps and recording stack and register maps at every instruction and so far the results are quite promising.
/cc @drchase @RLH @randall77 @minux
Activity
gopherbot commentedon Mar 26, 2018
Change https://golang.org/cl/102600 mentions this issue:
design: add 24543-non-cooperative-preemption
gopherbot commentedon Mar 26, 2018
Change https://golang.org/cl/102603 mentions this issue:
cmd/compile: detect simple inductive facts in prove
gopherbot commentedon Mar 26, 2018
Change https://golang.org/cl/102604 mentions this issue:
cmd/compile: don't produce a past-the-end pointer in range loops
aclements commentedon Mar 27, 2018
Forwarding some questions from @hyangah on the CL:
All of cgo is currently considered a safe-point (one of the reasons it's relatively expensive to enter and exit cgo) and this won't change.
I don't think the runtime can avoid sending signals to threads that may be in cgo without expensive synchronization on common paths, but I don't think it matters. When it enters the runtime signal handler it can recognize that it was in cgo and do the appropriate thing (which will probably be to just ignore it, or maybe queue up an action like stack scanning).
It should be okay if cgo code uses the signal, as long as it's correctly chained. I'm hoping to use POSIX real-time signals on systems where they're available, so the runtime will attempt to find one that's unused (which is usually all of them anyway), though that isn't an option on Darwin.
And a question from @randall77 (which I answered on the CL, but should have answered here):
There's really no cost to the current technique and we'll continue to rely on it in the runtime for the foreseeable future, so my current plan is to leave it in. However, we could be much more aggressive about removing stack bounds checks (for example if we can prove that a whole call tree will fit in the nosplit zone).
TocarIP commentedon Mar 27, 2018
So it is still possible to make goroutine nonpreemptable with something like:
sha256.Sum(make([]byte,1000000000))
where inner loop is written in asm?
aclements commentedon Mar 27, 2018
Yes, that would still make a goroutine non-preemptible. However, with some extra annotations in the assembly to indicate registers containing pointers it will become preemptible without any extra work or run-time overhead to reach an explicit safe-point. In the case of
sha256.Sum
these annotations would probably be trivial since it will never construct a pointer that isn't shadowed by the arguments (so it can claim there are no pointers in registers).I'll add a paragraph to the design doc about this.
komuw commentedon Mar 28, 2018
will the design doc be posted here?
aclements commentedon Mar 28, 2018
design: add 24543-non-cooperative-preemption
aclements commentedon Mar 28, 2018
The doc is now submitted: Proposal: Non-cooperative goroutine preemption
mtstickney commentedon Mar 30, 2018
Disclaimer: I'm not a platform expert, or an expert on language implementations, or involved with go aside from having written a few toy programs in it. That said:
There's a (potentially) fatal flaw here:
GetThreadContext
doesn't actually work on Windows (see here for details). There are several lisp implementations that have exhibited crashes on that platform because they tried to useGetThreadContext
/SetThreadContext
to implement preemptive signals on Windows.As some old notes for SBCL point out, Windows has no working version of preemptive signals without loading a kernel driver, which is generally prohibitive for applications.
JamesBielby commentedon Mar 31, 2018
I think the example code to avoid creating a past-the-end pointer has a problem if the slice has a capacity of 0. You need to declare _p after the first if statement.
205 remaining items
networkimprov commentedon Jul 3, 2020
@szmcdull have you tried this runtime switch? Note that Go had preemption before 1.14...
szmcdull commentedon Jul 3, 2020
Yes I tried. But still got
panic: concurrent map iteration and map
networkimprov commentedon Jul 3, 2020
I think you were just lucky that you didn't see that before 1.14 :-)
Try https://golang.org/pkg/sync/#Map
For further Q's, I refer you to golang-nuts. You'll get more & faster responses there, generally.
seccomp: Add wait_killable semantic to seccomp user notifier
seccomp: Add wait_killable semantic to seccomp user notifier
seccomp: Add wait_killable semantic to seccomp user notifier
seccomp: Add wait_killable semantic to seccomp user notifier
seccomp: Add wait_killable semantic to seccomp user notifier