Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
One of the common OOM issue in Pig is, Pig hitting OOM while trying to spill a large bag. Current solutions is to give higher heapsize or tweak
pig.spill.memory.usage.threshold.fraction pig.spill.collection.threshold.fraction pig.spill.unused.memory.threshold.size
and make sure spilling starts early enough. These params are still critical but wondering if any improvement can be made to increase the chances of avoiding OOM while spilling a single large bag.