[DRILL-6535] ClassCastException in Lateral Unnest queries when dealing with schema changed json data

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.14.0
• Component/s: Execution - Relational Operators
• Labels: ready-to-commit
• Assignee: Sorabh Hamirwasia
• Reporter: Sorabh Hamirwasia
• Reviewer: Parth Chandra
• Created: 25/Jun/18 16:24
• Updated: 02/Jul/18 23:37
• Resolved: 02/Jul/18 23:37

Description

Found by kedarbcs16

ClassCastException in Lateral Unnest queries when dealing with schema changed json data

Query : SELECT  customer.c_custkey,customer.c_acctbal,orders.o_orderkey, orders.o_totalprice,orders.o_orderdate FROM customer, LATERAL 
(SELECT O.ord.o_orderkey as o_orderkey, O.ord.o_totalprice as o_totalprice,O.ord.o_orderdate as o_orderdate  FROM UNNEST(customer.c_orders) O(ord) WHERE year(O.ord.o_orderdate) <> 1998)orders;

The data is sf001 complex data in multi json format partitioned based on the year of the o_orderdata column.
The last json file(for year 1998) has 2 schema changes for c_acctbal and o_shippriority .
The logs are :-

[Error Id: 6df4ceae-c989-4592-aeec-6d30b626f0ab on drill182:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: ClassCastException: org.apache.drill.exec.vector.NullableFloat8Vector cannot be cast to org.apache.drill.exec.vector.NullableBigIntVector

Fragment 0:0

[Error Id: 6df4ceae-c989-4592-aeec-6d30b626f0ab on drill182:31010]
        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:359) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:214) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:325) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.ClassCastException: org.apache.drill.exec.vector.NullableFloat8Vector cannot be cast to org.apache.drill.exec.vector.NullableBigIntVector
        at org.apache.drill.exec.vector.NullableBigIntVector.copyEntry(NullableBigIntVector.java:396) ~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.copyDataToOutputVectors(LateralJoinBatch.java:802) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.emitLeft(LateralJoinBatch.java:813) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.crossJoinAndOutputRecords(LateralJoinBatch.java:761) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.produceOutputBatch(LateralJoinBatch.java:479) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.innerNext(LateralJoinBatch.java:157) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:137) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:137) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:292) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:279) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_161]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0.jar:na]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:279) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
        ... 4 common frames omitted

As discussed with shamirwasia the bug is that the subsequent left incoming batch has a schema change when the output batch is having 0 elements because of filter.

Issue Links

• is depended upon by: DRILL-5999 Add support for LATERAL join
• links to: GitHub Pull Request #1339