[PIG-5083] CombinerPackager and LitePackager should not materialize bags

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 06/Jan/17 21:34
• Updated: 21/Jun/17 09:15
• Resolved: 18/Jan/17 13:38

Description

Before PIG-3591 and creation of CombinerPackager, POCombinerPackage directly read from the combiner/reducer input instead of materializing the bag.

https://github.com/apache/pig/blob/branch-0.12/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java#L140-L161

The unnecessary materialization leads to lot of spills and OOMs in some cases.

Issue Links

• is broken by: PIG-3591 Refactor POPackage to separate MR specific code from packaging