[HIVE-17896] TopNKey: Create a standalone vectorizable TopNKey operator

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: Operators
• Assignee: Teddy Choi
• Reporter: Gopal Vijayaraghavan
• Created: 24/Oct/17 23:21
• Updated: 27/Feb/24 22:23
• Resolved: 19/Jul/18 21:24

Description

For TPC-DS Query27, the TopN operation is delayed by the group-by - the group-by operator buffers up all the rows before discarding the 99% of the rows in the TopN Hash within the ReduceSink Operator.

The RS TopN operator is very restrictive as it only supports doing the filtering on the shuffle keys, but it is better to do this before breaking the vectors into rows and losing the isRepeating properties.

Adding a TopN Key operator in the physical operator tree allows the following to happen.

GBY->RS(Top=1)

can become

TNK(1)>GBY>RS(Top=1)

So that, the TopNKey can remove rows before they are buffered into the GBY and consume memory.

Here's the equivalent implementation in Presto

https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35

Adding this as a sub-feature of GroupBy prevents further optimizations if the GBY is on keys "a,b,c" and the TopNKey is on just "a".

Issue Links

• is related to: HIVE-22749 ReEnable TopNKey optimization in vectorized q tests
• is related to: HIVE-23101 Fix topnkey_grouping_sets
• is related to: HIVE-23120 TopNKey related tests should be run by TestMiniLlapLocalCliDriver only
• is related to: HIVE-20368 Remove VectorTopNKeyOperator lock
• is related to: HIVE-22435 Exception when using VectorTopNKeyOperator operator
• is related to: HIVE-22726 TopN Key optimizer should use array instead of priority queue
• is related to: HIVE-22925 Implement TopNKeyFilter efficiency check
• is related to: HIVE-22982 TopN Key efficiency check might disable filter too soon
• is related to: HIVE-20150 TopNKey pushdown
• is related to: HIVE-22464 Implement support for NULLS FIRST/LAST in TopNKeyOperator
• is related to: HIVE-22652 TopNKey push through Group by with Grouping sets
• is related to: HIVE-22666 Introduce TopNKey operator for PTF Reduce Sink
• is related to: HIVE-22735 TopNKey operator deduplication
• is related to: HIVE-22867 Add partitioning support to VectorTopNKeyOperator
• is related to: HIVE-23736 Disable topn in ReduceSinkOp if a TNK is introduced
• relates to: HIVE-22507 KeyWrapper comparator create field comparator instances at every comparison
• links to: Review Board