[FLINK-29717] Supports hive udaf such as sum/count by native implementation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.20.0
• Component/s: Connectors / Hive, Table SQL / Runtime
• Assignee: Unassigned
• Reporter: dalongliu
• Created: 21/Oct/22 10:22
• Updated: 11/Mar/24 12:44

Description

The current Flink side of the Hive UDAF has a unified encapsulation HiveGenericUDAF, and the intermediate result type of the aggregation function is encapsulated as RAW type, which is a variable-length data type that is serialized and deserialized by default using the Kryo serializer, so BinaryRowData does not support in-place updates to this type, which also leads to aggregation function that uses the RAW type as an intermediate aggregation buffer and cannot use hash-based aggregation strategies. Since the intermediate state type of Hive's UDAF is RAW type, it also cannot use hash-based aggregation strategy, and the overall performance in TPC-DS scenario is more than 2 times than the performance of Flink's built-in function. After some research, here we propose implementing this common udaf in a native way.

Issue Links

• duplicates: FLINK-25641 Unexpected aggregate plan after load hive module

Sub-Tasks

• Supports hive sum function by native implementation: Closed
• Supports hive count function by native implementation: Closed
• Supports hive average function by native implemetatoin: Closed
• Supports hive min function by native implementation: Closed
• Supports hive max function by native implementation: Closed
• Supports hive first_value function by native implementation: Open
• Supports hive last_value function by native implementation: Open
• Supports hive std function by native implementation: Open
• Supports hive stddev_samp function by native implementation: Open
• Fix the bug of sum(try_cast(string as bigint)) return null when partial elements can't convert to bigint: Closed
• Introducing cofigurable option to enable hive native function: Closed
• Add document for new introduced hive agg option: Closed
• The option `table.exec.hive.native-agg-function.enabled` should work at job level when using it in SqlClient side: Open
• Fix sum function can't get the corrected value when the argument type is string: Closed