[PIG-3044] Trigger POPartialAgg compaction under GC pressure

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0, 0.11, 0.10.1
• Fix Version/s: 0.11, 0.12.0

Release Note

When pig.exec.mapPartAgg, previously the memory reserved for in-memory aggregation was roughly 1/3 the value of pig.cachedbag.memusage per POPartialAgg operator. Reserving that much RAM could cause problems for jobs that required a lot of memory. POPartialAgg is now spillable, and regardless of the setting of pig.cachedbag.memusage, will happily shrink when memory is needed.

• Assignee: Dmitriy V. Ryaboy
• Reporter: Dmitriy V. Ryaboy
• Created: 09/Nov/12 22:39
• Updated: 22/Feb/13 04:53
• Resolved: 10/Dec/12 21:26

Description

If partial aggregation is turned on in pig 10 and 11, 20% (by default) of the available heap can be consumed by the POPartialAgg operator. This can cause memory issues for jobs that use all, or nearly all, of the heap already.

If we make POPartialAgg "spillable" (trigger compaction when memory reduction is required), we would be much nicer to high-memory jobs.