[PIG-4237] Error when there is a bag inside an RDD

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Labels: spork
• Patch Info: Patch Available
• Assignee: Carlos Balduz
• Reporter: Carlos Balduz
• Created: 15/Oct/14 10:59
• Updated: 21/Jun/17 09:18
• Resolved: 20/Oct/14 11:16

Description

Bags cannot be sent to an RDD, as it produces a SelfSpillBag$MemoryLimits not Serializable exception. This results in an error for almost every operation performed after grouping tuples.

This error is fixed after making transient the protected MemoryLimit memLimit attribute inside org.apache.pig.data.SelfSpillBag, but I do not know the impact of this change.

Issue Links

• blocks: PIG-4234 Order By error after Group By in Spark