Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5694

hash agg spill to disk, second phase OOM

    Details

      Description

      1.11.0-SNAPSHOT d622f76ee6336d97c9189fc589befa7b0f4189d6 DRILL-5165: For limit all case, no need to push down limit to scan 21.07.2017 @ 10:36:29 PDT

      Second phase agg ran out of memory. Not suppose to. Test data currently only accessible locally.

      /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q
      Query:
      select row_count, sum(row_count), avg(double_field), max(double_rand), count(float_rand) from parquet_500m_v1 group by row_count order by row_count limit 30
      Failed with exception
      java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

      HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far allocated: 534773760.
      Fragment 1:6

      [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010]

      (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far allocated: 534773760.
      org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175
      org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
      org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.record.AbstractRecordBatch.next():119
      org.apache.drill.exec.record.AbstractRecordBatch.next():109
      org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
      org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.record.AbstractRecordBatch.next():119
      org.apache.drill.exec.record.AbstractRecordBatch.next():109
      org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.record.AbstractRecordBatch.next():119
      org.apache.drill.exec.record.AbstractRecordBatch.next():109
      org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
      org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.physical.impl.BaseRootExec.next():105
      org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
      org.apache.drill.exec.physical.impl.BaseRootExec.next():95
      org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
      org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
      java.security.AccessController.doPrivileged():-2
      javax.security.auth.Subject.doAs():415
      org.apache.hadoop.security.UserGroupInformation.doAs():1595
      org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
      org.apache.drill.common.SelfCleaningRunnable.run():38
      java.util.concurrent.ThreadPoolExecutor.runWorker():1145
      java.util.concurrent.ThreadPoolExecutor$Worker.run():615
      java.lang.Thread.run():745
      Caused By (org.apache.drill.exec.exception.OutOfMemoryException) Unable to allocate buffer of size 4194304 due to memory limit. Current allocation: 534773760
      org.apache.drill.exec.memory.BaseAllocator.buffer():238
      org.apache.drill.exec.memory.BaseAllocator.buffer():213
      org.apache.drill.exec.vector.IntVector.allocateBytes():231
      org.apache.drill.exec.vector.IntVector.allocateNew():211
      org.apache.drill.exec.test.generated.HashTableGen2141.allocMetadataVector():778
      org.apache.drill.exec.test.generated.HashTableGen2141.resizeAndRehashIfNeeded():717
      org.apache.drill.exec.test.generated.HashTableGen2141.insertEntry():643
      org.apache.drill.exec.test.generated.HashTableGen2141.put():618
      org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1173
      org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
      org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.record.AbstractRecordBatch.next():119
      org.apache.drill.exec.record.AbstractRecordBatch.next():109
      org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
      org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.record.AbstractRecordBatch.next():119
      org.apache.drill.exec.record.AbstractRecordBatch.next():109
      org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.record.AbstractRecordBatch.next():119
      org.apache.drill.exec.record.AbstractRecordBatch.next():109
      org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
      org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
      org.apache.drill.exec.record.AbstractRecordBatch.next():162
      org.apache.drill.exec.physical.impl.BaseRootExec.next():105
      org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
      org.apache.drill.exec.physical.impl.BaseRootExec.next():95
      org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
      org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
      java.security.AccessController.doPrivileged():-2
      javax.security.auth.Subject.doAs():415
      org.apache.hadoop.security.UserGroupInformation.doAs():1595
      org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
      org.apache.drill.common.SelfCleaningRunnable.run():38
      java.util.concurrent.ThreadPoolExecutor.runWorker():1145
      java.util.concurrent.ThreadPoolExecutor$Worker.run():615
      java.lang.Thread.run():745

      at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
      at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
      at oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
      at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
      at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:220)
      at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:101)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

      HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far allocated: 534773760.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ben-zvi Boaz Ben-Zvi
                Reporter:
                cchang@maprtech.com Chun Chang
                Reviewer:
                Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: