[FLINK-1621] Create a generalized combine function

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9
• Fix Version/s: 0.9
• Component/s: Runtime / Coordination
• Labels: features performance
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 02/Mar/15 09:38
• Updated: 01/Apr/15 10:24
• Resolved: 18/Mar/15 17:51

Description

Flink allows combiners which accept a type I and combine the values of this type into type O. In Google Dataflow, combiners are more generalized. They accept an Input I, produce an intermediate combine value of T, and finally an output O. Flink's combiners are like the SimpleCombineFn in Google Dataflow.

Right now, we translate the KeyedCombineFn into a SortPartition followed by a MapPartition to emulate the Combiner's behavior. Rudimentary performance tests showed that this behavior causes a significant increase in run time compared to the proper Combine implementation.

Let's implement a more generalized Combiner to create a better mapping from Google Dataflow to Flink.

Issue Links

• is related to: FLINK-1780 Rename FlatCombineFunction to GroupCombineFunction
• relates to: FLINK-1043 Alternative combine interface

Sub-Tasks

• Add GroupReducePartial Operator: Resolved