[PIG-4802] Autoparallelism should estimate less when there is combiner

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 10/Feb/16 14:52
• Updated: 08/Jun/16 20:48
• Resolved: 10/Feb/16 21:23

Description

When there is a combiner, it reduces records by a lot. Auto-parallelism should take that into account.

Also currently we multiply by a factor of 10 if there is any FLATTEN. Users usually have FLATTEN(group) when the group by key is a compound key and it ends up estimating high. Only FLATTEN of bag should be considered.