[FLINK-5722] Implement DISTINCT as dedicated operator

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.3.0
• Fix Version/s: 1.3.0
• Component/s: Table SQL / API
• Assignee: Fabian Hueske
• Reporter: Fabian Hueske
• Created: 06/Feb/17 13:55
• Updated: 08/Mar/17 13:16
• Resolved: 08/Mar/17 13:16

Description

DISTINCT is currently implemented for batch Table API / SQL as an aggregate which groups on all fields. Grouped aggregates are implemented as GroupReduce with sort-based combiner.

This operator can be more efficiently implemented by using ReduceFunction and hinting a HashCombine strategy. The same ReduceFunction can be used for all DISTINCT operations and can be assigned with appropriate forward field annotations.

We would need a custom conversion rule which translates distinct aggregations (grouping on all fields and returning all fields) into a custom DataSetRelNode.

Issue Links

• links to: GitHub Pull Request #3471