[DRILL-5670] Varchar vector throws an assertion error when allocating a new vector

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.11.0
• Component/s: Execution - Relational Operators
• Assignee: Paul Rogers
• Reporter: Rahul Kumar Challapalli
• Created: 13/Jul/17 00:49
• Updated: 01/Jun/18 20:29

Description

I am running this test on a private branch of paul's repository. Below is the commit info

git.commit.id.abbrev=d86e16c
git.commit.user.email=[e-mail]
git.commit.message.full=DRILL-5601\: Rollup of external sort fixes an improvements\n\n- DRILL-5513\: Managed External Sort \: OOM error during the merge phase\n- DRILL-5519\: Sort fails to spill and results in an OOM\n- DRILL-5522\: OOM during the merge and spill process of the managed external sort\n- DRILL-5594\: Excessive buffer reallocations during merge phase of external sort\n- DRILL-5597\: Incorrect "bits" vector allocation in nullable vectors allocateNew()\n- DRILL-5602\: Repeated List Vector fails to initialize the offset vector\n\nAll of the bugs have to do with handling low-memory conditions, and with\ncorrectly estimating the sizes of vectors, even when those vectors come\nfrom the spill file or from an exchange. Hence, the changes for all of\nthe above issues are interrelated.\n
git.commit.id=d86e16c551e7d3553f2cde748a739b1c5a7a7659
git.commit.message.short=DRILL-5601\: Rollup of external sort fixes an improvements
git.commit.user.name=Paul Rogers
git.build.user.name=Rahul Challapalli
git.commit.id.describe=0.9.0-1078-gd86e16c
git.build.user.email=[e-mail]
git.branch=d86e16c551e7d3553f2cde748a739b1c5a7a7659
git.commit.time=05.07.2017 @ 20\:34\:39 PDT
git.build.time=12.07.2017 @ 14\:27\:03 PDT
git.remote.origin.url=[e-mail]\:paul-rogers/drill.git

Below query fails with an Assertion Error

0: jdbc:drill:zk=10.10.100.190:5181> ALTER SESSION SET `exec.sort.disable_managed` = false;
+-------+-------------------------------------+
|  ok   |               summary               |
+-------+-------------------------------------+
| true  | exec.sort.disable_managed updated.  |
+-------+-------------------------------------+
1 row selected (1.044 seconds)
0: jdbc:drill:zk=10.10.100.190:5181> alter session set `planner.memory.max_query_memory_per_node` = 482344960;
+-------+----------------------------------------------------+
|  ok   |                      summary                       |
+-------+----------------------------------------------------+
| true  | planner.memory.max_query_memory_per_node updated.  |
+-------+----------------------------------------------------+
1 row selected (0.372 seconds)
0: jdbc:drill:zk=10.10.100.190:5181> alter session set `planner.width.max_per_node` = 1;
+-------+--------------------------------------+
|  ok   |               summary                |
+-------+--------------------------------------+
| true  | planner.width.max_per_node updated.  |
+-------+--------------------------------------+
1 row selected (0.292 seconds)
0: jdbc:drill:zk=10.10.100.190:5181> alter session set `planner.width.max_per_query` = 1;
+-------+---------------------------------------+
|  ok   |                summary                |
+-------+---------------------------------------+
| true  | planner.width.max_per_query updated.  |
+-------+---------------------------------------+
1 row selected (0.25 seconds)
0: jdbc:drill:zk=10.10.100.190:5181> select count(*) from (select * from dfs.`/drill/testdata/resource-manager/3500cols.tbl` order by columns[450],columns[330],columns[230],columns[220],columns[110],columns[90],columns[80],columns[70],columns[40],columns[10],columns[20],columns[30],columns[40],columns[50], columns[454],columns[413],columns[940],columns[834],columns[73],columns[140],columns[104],columns[2222],columns[30],columns[2420],columns[1520], columns[1410], columns[1110],columns[1290],columns[2380],columns[705],columns[45],columns[1054],columns[2430],columns[420],columns[404],columns[3350], columns[3333],columns[153],columns[356],columns[84],columns[745],columns[1450],columns[103],columns[2065],columns[343],columns[3420],columns[530], columns[3210] ) d where d.col433 = 'sjka skjf';
Error: RESOURCE ERROR: External Sort encountered an error while spilling to disk

Fragment 2:0

[Error Id: 26b55576-1a5c-4756-96d6-fbec25eecf03 on qa-node190.qa.lab:31010]

  (java.lang.AssertionError) null
    org.apache.drill.exec.vector.VarCharVector.allocateNew():400
    org.apache.drill.exec.vector.RepeatedVarCharVector.allocateNew():272
    org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount():37
    org.apache.drill.exec.vector.AllocationHelper.allocate():44
    org.apache.drill.exec.record.SmartAllocationHelper.allocateVector():117
    org.apache.drill.exec.record.SmartAllocationHelper.allocateVector():97
    org.apache.drill.exec.record.SmartAllocationHelper.allocateBatch():87
    org.apache.drill.exec.physical.impl.xsort.managed.PriorityQueueCopierWrapper$BatchMerger.next():262
    org.apache.drill.exec.physical.impl.xsort.managed.SpilledRuns.doMergeAndSpill():171
    org.apache.drill.exec.physical.impl.xsort.managed.SpilledRuns.safeMergeAndSpill():138
    org.apache.drill.exec.physical.impl.xsort.managed.SpilledRuns.mergeAndSpill():91
    org.apache.drill.exec.physical.impl.xsort.managed.SortImpl.spillFromMemory():298
    org.apache.drill.exec.physical.impl.xsort.managed.SortImpl.addBatch():191
    org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():402
    org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():338
    org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():290
    org.apache.drill.exec.record.AbstractRecordBatch.next():162
    org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():225
    org.apache.drill.exec.record.AbstractRecordBatch.next():119
    org.apache.drill.exec.record.AbstractRecordBatch.next():109
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
    org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
    org.apache.drill.exec.record.AbstractRecordBatch.next():162
    org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():225
    org.apache.drill.exec.physical.impl.BaseRootExec.next():105
    org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
    org.apache.drill.exec.physical.impl.BaseRootExec.next():95
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():415
    org.apache.hadoop.security.UserGroupInformation.doAs():1595
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745 (state=,code=0)

The data set is too large to attach here. Logs and profiles are attached

Issue Links

• Is contained by: DRILL-5601 Rollup of External Sort memory management fixes