Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12084

Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      STEPS TO REPRODUCE:

      CREATE TABLE `sample_07` ( `code` string , `description` string , `total_emp` int , `salary` int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TextFile;
      load data local inpath 'sample_07.csv'  into table sample_07;
      set hive.limit.pushdown.memory.usage=0.9999;
      select * from sample_07 order by salary LIMIT 999999999;
      

      This will result in

      Caused by: java.lang.OutOfMemoryError: Java heap space
      	at org.apache.hadoop.hive.ql.exec.TopNHash.initialize(TopNHash.java:113)
      	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initializeOp(ReduceSinkOperator.java:234)
      	at org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:68)
      	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
      	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
      	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
      

      The basic issue lies with top n optimization. We need a limit for the top n optimization. Ideally we would detect that the allocated bytes will be bigger than the "limit.pushdown.memory.usage" without trying to alloc it.

        Attachments

        1. HIVE-12084.4.patch
          3 kB
          Hari Sankar Sivarama Subramaniyan
        2. HIVE-12084.3.patch
          14 kB
          Hari Sankar Sivarama Subramaniyan
        3. HIVE-12084.2.patch
          8 kB
          Hari Sankar Sivarama Subramaniyan
        4. HIVE-12084.1.patch
          8 kB
          Hari Sankar Sivarama Subramaniyan

          Issue Links

            Activity

              People

              • Assignee:
                hsubramaniyan Hari Sankar Sivarama Subramaniyan
                Reporter:
                hsubramaniyan Hari Sankar Sivarama Subramaniyan
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: