Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
The current Flink side of the Hive UDAF has a unified encapsulation HiveGenericUDAF, and the intermediate result type of the aggregation function is encapsulated as RAW type, which is a variable-length data type that is serialized and deserialized by default using the Kryo serializer, so BinaryRowData does not support in-place updates to this type, which also leads to aggregation function that uses the RAW type as an intermediate aggregation buffer and cannot use hash-based aggregation strategies. Since the intermediate state type of Hive's UDAF is RAW type, it also cannot use hash-based aggregation strategy, and the overall performance in TPC-DS scenario is more than 2 times than the performance of Flink's built-in function. After some research, here we propose implementing this common udaf in a native way.
Attachments
Attachments
Issue Links
- duplicates
-
FLINK-25641 Unexpected aggregate plan after load hive module
- Resolved