[DRILL-6384] TPC-H tests fail with OOM - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Duplicate
Affects Version/s: 1.14.0
Fix Version/s: None
Component/s: Execution - Flow
Labels:
None

Description

On latest Apache master, we are observing that there are multiple test failures. It looks like Drill runs out of Direct memory and queries fail with OOM. Few other queries fail probably fail because they are unable to connect to Drillbits.

It looks like one of the recent commits caused this.

Commit ID	Status
24193b1b038a6315681a65c76a67034b64f71fc5	FAIL
883c8d94b0021a83059fa79563dd516c4299b70a	FAIL
2601cdd33e0685f59a7bf2ac72541bd9dcaaa18f	FAIL
9173308710c3decf8ff745493ad3e85ccdaf7c37	PASS
c6549e58859397c88cb1de61b4f6eee52a07ed0c	PASS

Two example queries + exceptions below. Also query log attached.

Query 1: Advanced/tpch/tpch_sf100/parquet/10.q

 select
 c.c_custkey,
 c.c_name,
 sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
 c.c_acctbal,
 n.n_name,
 c.c_address,
 c.c_phone,
 c.c_comment
 from
 customer c,
 orders o,
 lineitem l,
 nation n
 where
 c.c_custkey = o.o_custkey
 and l.l_orderkey = o.o_orderkey
 and o.o_orderdate >= date '1994-03-01'
 and o.o_orderdate < date '1994-03-01' + interval '3' month
 and l.l_returnflag = 'R'
 and c.c_nationkey = n.n_nationkey
 group by
 c.c_custkey,
 c.c_name,
 c.c_acctbal,
 c.c_phone,
 n.n_name,
 c.c_address,
 c.c_comment
 order by
 revenue desc
 limit 20

Exception:

java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

AGGR OOM at First Phase. Partitions: 8. Estimated batch size: 18481152. values size: 1048576. Output alloc size: 1048576. Planned batches: 8 Memory limit: 313709266 so far allocated: 2097152. 
 Fragment 4:88

[Error Id: 81017b59-dfa3-4db9-8673-bee7b80f8acd on atsqa6c82.qa.lab:31010]

(org.apache.drill.exec.exception.OutOfMemoryException) AGGR OOM at First Phase. Partitions: 8. Estimated batch size: 18481152. values size: 1048576. Output alloc size: 1048576. Planned batches: 8 Memory limit: 313709266 so far allocated: 2097152. 
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.spillIfNeeded():1419
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.doSpill():1381
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.checkGroupAndAggrValues():1304
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.doWork():592
 org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
 org.apache.drill.exec.record.AbstractRecordBatch.next():164
 org.apache.drill.exec.physical.impl.BaseRootExec.next():105
 org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():152
 org.apache.drill.exec.physical.impl.BaseRootExec.next():95
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
 java.security.AccessController.doPrivileged():-2
 javax.security.auth.Subject.doAs():422
 org.apache.hadoop.security.UserGroupInformation.doAs():1595
 org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
 org.apache.drill.common.SelfCleaningRunnable.run():38
 java.util.concurrent.ThreadPoolExecutor.runWorker():1149
 java.util.concurrent.ThreadPoolExecutor$Worker.run():624
 java.lang.Thread.run():748

at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:530)
 at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:634)
 at oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:207)
 at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:155)
 at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
 at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
 Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

AGGR OOM at First Phase. Partitions: 8. Estimated batch size: 18481152. values size: 1048576. Output alloc size: 1048576. Planned batches: 8 Memory limit: 313709266 so far allocated: 2097152. 
 Fragment 4:88

[Error Id: 81017b59-dfa3-4db9-8673-bee7b80f8acd on atsqa6c82.qa.lab:31010]

(org.apache.drill.exec.exception.OutOfMemoryException) AGGR OOM at First Phase. Partitions: 8. Estimated batch size: 18481152. values size: 1048576. Output alloc size: 1048576. Planned batches: 8 Memory limit: 313709266 so far allocated: 2097152. 
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.spillIfNeeded():1419
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.doSpill():1381
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.checkGroupAndAggrValues():1304
 org.apache.drill.exec.test.generated.HashAggregatorGen3296.doWork():592
 org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
 org.apache.drill.exec.record.AbstractRecordBatch.next():164
 org.apache.drill.exec.physical.impl.BaseRootExec.next():105
 org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():152
 org.apache.drill.exec.physical.impl.BaseRootExec.next():95
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
 java.security.AccessController.doPrivileged():-2
 javax.security.auth.Subject.doAs():422
 org.apache.hadoop.security.UserGroupInformation.doAs():1595
 org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
 org.apache.drill.common.SelfCleaningRunnable.run():38
 java.util.concurrent.ThreadPoolExecutor.runWorker():1149
 java.util.concurrent.ThreadPoolExecutor$Worker.run():624
 java.lang.Thread.run():748

Query 2:
Advanced/tpch/tpch_sf100/parquet/08.q

select
o_year,
sum(case
when nation = 'EGYPT' then volume
else 0
end) / sum(volume) as mkt_share
from
(
select
extract(year from o.o_orderdate) as o_year,
l.l_extendedprice * (1 - l.l_discount) as volume,
n2.n_name as nation
from
part p,
supplier s,
lineitem l,
orders o,
customer c,
nation n1,
nation n2,
region r
where
p.p_partkey = l.l_partkey
and s.s_suppkey = l.l_suppkey
and l.l_orderkey = o.o_orderkey
and o.o_custkey = c.c_custkey
and c.c_nationkey = n1.n_nationkey
and n1.n_regionkey = r.r_regionkey
and r.r_name = 'MIDDLE EAST'
and s.s_nationkey = n2.n_nationkey
and o.o_orderdate between date '1995-01-01' and date '1996-12-31'
and p.p_type = 'PROMO BRUSHED COPPER'
) as all_nations
group by
o_year
order by
o_year

Exception:

java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

Failure allocating buffer.
Fragment 4:57

[Error Id: a5eeae54-ac8f-42fa-9af1-03247e6bc316 on atsqa6c82.qa.lab:31010]

  (org.apache.drill.exec.exception.OutOfMemoryException) Failure allocating buffer.
    io.netty.buffer.PooledByteBufAllocatorL.allocate():67
    org.apache.drill.exec.memory.AllocationManager.<init>():84
    org.apache.drill.exec.memory.BaseAllocator.bufferWithoutReservation():258
    org.apache.drill.exec.memory.BaseAllocator.buffer():241
    org.apache.drill.exec.memory.BaseAllocator.buffer():211
    org.apache.drill.exec.vector.VarCharVector.allocateNew():389
    org.apache.drill.exec.vector.NullableVarCharVector.allocateNew():236
    org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount():41
    org.apache.drill.exec.vector.AllocationHelper.allocate():54
    org.apache.drill.exec.vector.AllocationHelper.allocate():28
    org.apache.drill.exec.physical.impl.ScanBatch$Mutator.populateImplicitVectors():446
    org.apache.drill.exec.physical.impl.ScanBatch$Mutator.access$200():304
    org.apache.drill.exec.physical.impl.ScanBatch.populateImplicitVectorsAndSetCount():267
    org.apache.drill.exec.physical.impl.ScanBatch.next():175
    org.apache.drill.exec.record.AbstractRecordBatch.next():118
    org.apache.drill.exec.record.AbstractRecordBatch.next():108
    org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.record.AbstractRecordBatch.next():118
    org.apache.drill.exec.test.generated.HashJoinProbeGen4786.executeProbePhase():127
    org.apache.drill.exec.test.generated.HashJoinProbeGen4786.probeAndProject():235
    org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():220
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.record.AbstractRecordBatch.next():118
    org.apache.drill.exec.test.generated.HashJoinProbeGen4788.executeProbePhase():127
    org.apache.drill.exec.test.generated.HashJoinProbeGen4788.probeAndProject():235
    org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():220
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.physical.impl.BaseRootExec.next():105
    org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():152
    org.apache.drill.exec.physical.impl.BaseRootExec.next():95
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1595
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748
  Caused By (io.netty.util.internal.OutOfDirectMemoryError) failed to allocate 16777216 byte(s) of direct memory (used: 34359738368, max: 34359738368)
    io.netty.util.internal.PlatformDependent.incrementMemoryCounter():510
    io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner():464
    io.netty.buffer.PoolArena$DirectArena.allocateDirect():766
    io.netty.buffer.PoolArena$DirectArena.newChunk():742
    io.netty.buffer.PoolArena.allocateNormal():244
    io.netty.buffer.PoolArena.allocate():226
    io.netty.buffer.PoolArena.allocate():146
    io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL():169
    io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer():201
    io.netty.buffer.PooledByteBufAllocatorL.allocate():65
    org.apache.drill.exec.memory.AllocationManager.<init>():84
    org.apache.drill.exec.memory.BaseAllocator.bufferWithoutReservation():258
    org.apache.drill.exec.memory.BaseAllocator.buffer():241
    org.apache.drill.exec.memory.BaseAllocator.buffer():211
    org.apache.drill.exec.vector.VarCharVector.allocateNew():389
    org.apache.drill.exec.vector.NullableVarCharVector.allocateNew():236
    org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount():41
    org.apache.drill.exec.vector.AllocationHelper.allocate():54
    org.apache.drill.exec.vector.AllocationHelper.allocate():28
    org.apache.drill.exec.physical.impl.ScanBatch$Mutator.populateImplicitVectors():446
    org.apache.drill.exec.physical.impl.ScanBatch$Mutator.access$200():304
    org.apache.drill.exec.physical.impl.ScanBatch.populateImplicitVectorsAndSetCount():267
    org.apache.drill.exec.physical.impl.ScanBatch.next():175
    org.apache.drill.exec.record.AbstractRecordBatch.next():118
    org.apache.drill.exec.record.AbstractRecordBatch.next():108
    org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.record.AbstractRecordBatch.next():118
    org.apache.drill.exec.test.generated.HashJoinProbeGen4786.executeProbePhase():127
    org.apache.drill.exec.test.generated.HashJoinProbeGen4786.probeAndProject():235
    org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():220
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.record.AbstractRecordBatch.next():118
    org.apache.drill.exec.test.generated.HashJoinProbeGen4788.executeProbePhase():127
    org.apache.drill.exec.test.generated.HashJoinProbeGen4788.probeAndProject():235
    org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():220
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.physical.impl.BaseRootExec.next():105
    org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():152
    org.apache.drill.exec.physical.impl.BaseRootExec.next():95
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1595
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

drillbit.log.txt
04/May/18 01:19
37 kB
Abhishek Girish

Issue Links

duplicates

DRILL-6374 Transitive Closure leads to TPCH Queries regressions and OOM when run concurrency test

Resolved

Activity

People

Assignee:: Vitalii Diravka

Reporter:: Abhishek Girish

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 04/May/18 01:22

Updated:: 04/May/18 16:51

Resolved:: 04/May/18 16:51