I spent some time looking into a test failure of TestAsyncKuduSession.testInsertIntoUnavailableTablet(). The test fails if the machine is under load because of what looks something likke the following:
- the DeadlineTracker for a Batch is created with timeout 1
- the process loses the CPU for a few milliseconds
We get to the following:
Typically sendRpcToTablet() returns request.getDeferred(). But in the case that it has already timed out, it returns an error deferred immediately, without firing the request's errback. This means that the deferred in batchResponses contains an error, because the batch callbacks added by `addBatchCallbacks` don't actually get triggered.