Affects Version/s: None
Fix Version/s: None
Drill has long supported the OUT_OF_MEMORY iterator status. The idea is that an operator can realize it has encountered memory pressure and ask its downstream operator to free up some memory. However, an inspection of the code shows that the status is actually sent in only one place (UnorderedReceiverBatch), and then only in response to the operator hitting its allocator limit (which no other batch can do anything about.)
If an operator did choose to try to use this status, there are two key problems:
- The operator must be able to suspend itself at any point that it might need memory. For example, an operator that allocates a dozen vectors must be able to stop on, say, the 9th vector, then resume at that point on the subsequent call to next(). The complexity of the state machine needed to do this is very high.
- The downstream operators (who may not yet have seen rows) are the least likely to be able to release memory. It is the upstream operators (such as spillable operators) that might be able to spill some of the rows they are holding.
Presto suggests a nice alternative:
- An operator which encounters memory pressure asks the fragment executor for more memory.
- The fragment executor asks all other operators in that fragment to release memory if possible.
This allows a very simple memory recovery strategy:
Note that, since the fragment runs on a single thread, the above is simple to implement. Each operator is either idle (not executing) or in a call to next() on a child operator. These are both stable times to consider invoking spilling. Further, a sender could use this opportunity to write partially-filled batches to the network and release them rather than waiting for more data.
The only thing that can't be handled is, say, having an interior node flush a batch to its downstream operator in the same batch.
Proposed are two changes:
- Retire the OUT_OF_MEMORY status. Simply remove all references to it since it is never sent.
- Create a stub requestMemory() method in the operator context that does nothing now, but could be expanded to perform the work suggested above.