[DRILL-6254] IllegalArgumentException: the requested size must be non-negative

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.14.0
• Component/s: Execution - Flow
• Labels: ready-to-commit
• Assignee: Padma Penumarthy
• Reporter: Khurram Faraaz
• Reviewer: Karthikeyan Manivannan
• Created: 15/Mar/18 20:55
• Updated: 30/Mar/18 15:11
• Resolved: 30/Mar/18 15:11

Description

Flatten query fails due to IllegalArgumentException: the requested size must be non-negative.

Script to generate JSON data file is attached here.

0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS 
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > CAST( col_int AS INT) col_int, 
. . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint, 
. . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char, 
. . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256)) col_fxdln_str, 
. . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256)) col_varln_str, 
. . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float, 
. . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION) col_double, 
. . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date, 
. . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time, 
. . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp, 
. . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean, 
. . . . . . . . . . . . . . > col_binary, 
. . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
+-----------+----------------------------+
| Fragment | Number of records written |
+-----------+----------------------------+
| 0_0 | 9 |
+-----------+----------------------------+
1 row selected (0.29 seconds)

Reset all options and set slice_target=1
alter system reset all;
alter system set `planner.slice_target`=1;

output_batch_size was set to its default value
drill.exec.memory.operator.output_batch_size = 16777216

 

select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;

Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be non-negative

Fragment 0:0

[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010] (state=,code=0)
0: jdbc:drill:schema=dfs.tmp>

Stack trace from drillbit.log

2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet
2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max.
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 using 1 threads. Earliest start: 2.829000 μs, Latest start: 2.829000 μs, Average start: 2.829000 μs .
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 3 ms to read file metadata
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State to report: RUNNING
2018-03-15 12:19:44,905 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested RUNNING --> FAILED
2018-03-15 12:19:44,927 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested FAILED --> FINISHED
2018-03-15 12:19:44,928 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException: the requested size must be non-negative

Fragment 0:0

[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalArgumentException: the requested size must be non-negative

Fragment 0:0

[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
 at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:300) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalArgumentException: the requested size must be non-negative
 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) ~[guava-18.0.jar:na]
 at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:224) ~[drill-memory-base-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:211) ~[drill-memory-base-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.vector.BigIntVector.allocateBytes(BigIntVector.java:239) ~[vector-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.vector.BigIntVector.allocateNew(BigIntVector.java:219) ~[vector-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.vector.RepeatedBigIntVector.allocateNew(RepeatedBigIntVector.java:272) ~[vector-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.vector.AllocationHelper.allocatePrecomputedChildCount(AllocationHelper.java:41) ~[vector-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.vector.AllocationHelper.allocate(AllocationHelper.java:66) ~[vector-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.record.RecordBatchSizer$ColumnSize.allocateVector(RecordBatchSizer.java:403) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.doAlloc(FlattenRecordBatch.java:276) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.handleRemainder(FlattenRecordBatch.java:240) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext(FlattenRecordBatch.java:165) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
 at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_161]
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0-mapr-1707.jar:na]
 at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 ... 4 common frames omitted

Issue Links

• links to: GitHub Pull Request #1179