[PIG-3562] Implement combiner optimizations for DISTINCT

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: tez-branch
• Fix Version/s: tez-branch
• Component/s: tez
• Assignee: Alex Bain
• Reporter: Alex Bain
• Created: 05/Nov/13 19:34
• Updated: 21/Nov/14 05:59
• Resolved: 08/Jan/14 23:40

Description

Currently, DISTINCT is implemented in a straightforward manner per https://issues.apache.org/jira/browse/PIG-3538.

However, we can implement two types of combiner optimizations for DISTINCT, just as the MRCompiler does for map-reduce:
1. A simple DistinctCombiner that throws away the duplicate tuples
2. An optimizer that transforms certain uses of DISTINCT into an algebraic udf form