Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-29717

Supports hive udaf such as sum/count by native implementation

Agile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      The current Flink side of the Hive UDAF has a unified encapsulation HiveGenericUDAF, and the intermediate result type of the aggregation function is encapsulated as RAW type, which is a variable-length data type that is serialized and deserialized by default using the Kryo serializer, so BinaryRowData does not support in-place updates to this type, which also leads to aggregation function that uses the RAW type as an intermediate aggregation buffer and cannot use hash-based aggregation strategies. Since the intermediate state type of Hive's UDAF is RAW type, it also cannot use hash-based aggregation strategy, and the overall performance in TPC-DS scenario is more than 2 times than the performance of Flink's built-in function. After some research, here we propose implementing this common udaf in a native way.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            lsy dalongliu

            Dates

              Created:
              Updated:

              Slack

                Issue deployment