Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5824

1st phase Hash Aggregate allocates more memory than the limit

    Details

      Description

      The per query memory limit was set to 2G. But the 1st phase hash agg operator memory limit set was larger than that:

      AGGR OOM at First Phase. Partitions: 32. Estimated batch size: 4784128. values size: 3670016. Output alloc size: 3670016. Planned batches: 1 Memory limit: 2680684544 so far allocated: 374341632.
      Fragment 3:0
      [Error Id: b22fe6ad-b805-433c-bae7-c0f60c30bb99 on 10.10.30.168:31010]
      (org.apache.drill.exec.exception.OutOfMemoryException) AGGR OOM at First Phase. Partitions: 32. Estimated batch size: 4784128. values size: 3670016. Output alloc size: 3670016. Planned batches: 1 Memory limit: 2680684544 so far allocated: 374341632.
      org.apache.drill.exec.test.generated.HashAggregatorGen5265.checkGroupAndAggrValues():1350
      org.apache.drill.exec.test.generated.HashAggregatorGen5265.doWork():591
      org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():169
      org.apache.drill.exec.record.AbstractRecordBatch.next():164
      org.apache.drill.exec.record.AbstractRecordBatch.next():119
      org.apache.drill.exec.record.AbstractRecordBatch.next():109
      org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
      org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():141
      org.apache.drill.exec.record.AbstractRecordBatch.next():164
      org.apache.drill.exec.physical.impl.BaseRootExec.next():105
      org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
      org.apache.drill.exec.physical.impl.BaseRootExec.next():95
      org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
      org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
      java.security.AccessController.doPrivileged():-2
      javax.security.auth.Subject.doAs():415
      org.apache.hadoop.security.UserGroupInformation.doAs():1595
      org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
      org.apache.drill.common.SelfCleaningRunnable.run():38
      java.util.concurrent.ThreadPoolExecutor.runWorker():1145
      java.util.concurrent.ThreadPoolExecutor$Worker.run():615
      java.lang.Thread.run():745

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ben-zvi Boaz Ben-Zvi
                Reporter:
                ben-zvi Boaz Ben-Zvi
                Reviewer:
                Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 24h
                  24h
                  Remaining:
                  Remaining Estimate - 24h
                  24h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified