Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
incoming batches are currently queued up and held in memory without limit. If execution on a node is delayed, or is not able keep up with the rate of incoming batches, this could result in out of memory errors. We want to allow spooling incoming batches to disk once the total size of the batches in the queue reaches a threshold.