Major Description
The race is between calls to flushNotification.get() and inactiveBufferAvailable(). Suppose T1 calls inactiveBufferAvailable(), gets back false, but is descheduled before constructing a PleaseThrottleException. Now T2 is scheduled, finishes an outstanding flush, calls queueBuffer(), and resets flushNotification to an empty Deferred. When T1 is rescheduled, it throws a PTE with that empty Deferred.
What is the effect? If the user waits on the Deferred from the PTE, the user is effectively waiting on "the next flush", which, depending on the stream of operations, may take place soon, may not take place for some time, or may not take place at all.
To fix this, we should probably reorder the calls to flushNotification.get() in apply() to happen before calls to inactiveBufferAvailable(). That way, a race will yield a stale Deferred rather than an empty one, and waiting on the stale Deferred should be a no-op.