[HIVE-24207] LimitOperator can leverage ObjectCache to bail out quickly

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: László Bodor
• Reporter: Rajesh Balamohan
• Created: 29/Sep/20 06:18
• Updated: 04/Jun/24 08:59
• Resolved: 11/Dec/20 07:53

Description

select  ss_sold_date_sk from store_sales, date_dim where date_dim.d_year in (1998,1998+1,1998+2) and store_sales.ss_sold_date_sk = date_dim.d_date_sk limit 100;

 select distinct ss_sold_date_sk from store_sales, date_dim where date_dim.d_year in (1998,1998+1,1998+2) and store_sales.ss_sold_date_sk = date_dim.d_date_sk limit 100;

 

Queries like the above generate a large number of map tasks. Currently they don't bail out after generating enough amount of data.

It would be good to make use of ObjectCache & retain the number of records generated. LimitOperator/VectorLimitOperator can bail out for the later tasks in the operator's init phase itself.

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorLimitOperator.java#L57

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java#L58

Issue Links

• causes: HIVE-28281 Hive produces incorrect result when a vertex contains multiple LimitOperator
• causes: HIVE-25751 Ignore exceptions related to interruption when the limit is reached
• is blocked by: TEZ-4238 Check null mrReader in MRInput.close
• relates to: HIVE-25274 TestLimitOperator fails if default engine is Tez
• links to: GitHub Pull Request #1556