[DRILL-1799] kvgen() row limit

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.7.0
• Component/s: Functions - Drill
• Assignee: Chun Chang
• Reporter: Chun Chang
• Created: 02/Dec/14 02:40
• Updated: 28/Apr/15 01:18
• Resolved: 16/Dec/14 20:52

Description

#Mon Dec 01 11:15:02 PST 2014
git.commit.id.abbrev=a60e1db

Running the following query that contains kvgen() appears running out of memory and fails. There appears to be a 50 row limit. If I add a limit to the query, limit 50 or less works, with limit 51, the query fails.

0: jdbc:drill:schema=dfs.drillTestDir> select kvgen(sub.esr) from `es/esr-part` sub;

The query returns result but fails in the middle of displaying the result. No other error message except the following stack:

18:06:08.158 [2b82e176-0aeb-9b05-15f2-520eea269a1c:frag:0:0] WARN o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing fragment
java.lang.IllegalStateException: Attempted to close accountor with 151 buffer(s) still allocatedfor QueryId: 2b82e176-0aeb-9b05-15f2-520eea269a1c, MajorFragmentId: 0, MinorFragmentId: 0.

Total 1 allocation(s) of byte size(s): 4096, at stack location:
org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:212)
org.apache.drill.exec.vector.UInt1Vector.allocateNewSafe(UInt1Vector.java:137)
org.apache.drill.exec.vector.NullableBigIntVector.allocateNewSafe(NullableBigIntVector.java:173)
org.apache.drill.exec.vector.complex.MapVector.allocateNewSafe(MapVector.java:165)
org.apache.drill.exec.vector.complex.RepeatedMapVector.allocateNewSafe(RepeatedMapVector.java:236)
org.apache.drill.exec.vector.complex.MapVector.allocateNewSafe(MapVector.java:165)
org.apache.drill.exec.vector.complex.MapVector.allocateNewSafe(MapVector.java:165)
org.apache.drill.exec.vector.complex.impl.SingleMapWriter.allocate(SingleMapWriter.java:135)
org.apache.drill.exec.vector.complex.impl.RepeatedMapWriter.allocate(RepeatedMapWriter.java:137)
org.apache.drill.exec.vector.complex.impl.SingleListWriter.allocate(SingleListWriter.java:116)
org.apache.drill.exec.vector.complex.impl.ComplexWriterImpl.allocate(ComplexWriterImpl.java:157)
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doAlloc(ProjectRecordBatch.java:217)
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:144)
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:89)
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:129)
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:106)
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:124)
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:86)
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:76)
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:52)
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:129)
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:106)
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:124)
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67)