[HIVE-12084] Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 2.0.0
• Assignee: Hari Sankar Sivarama Subramaniyan
• Reporter: Hari Sankar Sivarama Subramaniyan
• Created: 09/Oct/15 23:32
• Updated: 16/Feb/16 23:50
• Resolved: 23/Oct/15 17:54

Description

STEPS TO REPRODUCE:

CREATE TABLE `sample_07` ( `code` string , `description` string , `total_emp` int , `salary` int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TextFile;
load data local inpath 'sample_07.csv'  into table sample_07;
set hive.limit.pushdown.memory.usage=0.9999;
select * from sample_07 order by salary LIMIT 999999999;

This will result in

Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.hadoop.hive.ql.exec.TopNHash.initialize(TopNHash.java:113)
	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initializeOp(ReduceSinkOperator.java:234)
	at org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:68)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)

The basic issue lies with top n optimization. We need a limit for the top n optimization. Ideally we would detect that the allocated bytes will be bigger than the "limit.pushdown.memory.usage" without trying to alloc it.

Issue Links

• relates to: HIVE-12310 Update memory estimation login in TopNHash
• links to: Review board