Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6535

ClassCastException in Lateral Unnest queries when dealing with schema changed json data

    Details

      Description

      Found by Kedar Sankar Behera

      ClassCastException in Lateral Unnest queries when dealing with schema changed json data

      Query : SELECT  customer.c_custkey,customer.c_acctbal,orders.o_orderkey, orders.o_totalprice,orders.o_orderdate FROM customer, LATERAL 
      (SELECT O.ord.o_orderkey as o_orderkey, O.ord.o_totalprice as o_totalprice,O.ord.o_orderdate as o_orderdate  FROM UNNEST(customer.c_orders) O(ord) WHERE year(O.ord.o_orderdate) <> 1998)orders;
      

      The data is sf001 complex data in multi json format partitioned based on the year of the o_orderdata column.
      The last json file(for year 1998) has 2 schema changes for c_acctbal and o_shippriority .
      The logs are :-

      [Error Id: 6df4ceae-c989-4592-aeec-6d30b626f0ab on drill182:31010]
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: ClassCastException: org.apache.drill.exec.vector.NullableFloat8Vector cannot be cast to org.apache.drill.exec.vector.NullableBigIntVector
      
      Fragment 0:0
      
      [Error Id: 6df4ceae-c989-4592-aeec-6d30b626f0ab on drill182:31010]
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:359) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:214) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:325) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
              at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
      Caused by: java.lang.ClassCastException: org.apache.drill.exec.vector.NullableFloat8Vector cannot be cast to org.apache.drill.exec.vector.NullableBigIntVector
              at org.apache.drill.exec.vector.NullableBigIntVector.copyEntry(NullableBigIntVector.java:396) ~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.copyDataToOutputVectors(LateralJoinBatch.java:802) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.emitLeft(LateralJoinBatch.java:813) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.crossJoinAndOutputRecords(LateralJoinBatch.java:761) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.produceOutputBatch(LateralJoinBatch.java:479) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.innerNext(LateralJoinBatch.java:157) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:137) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:137) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:292) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:279) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
              at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_161]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0.jar:na]
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:279) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
              ... 4 common frames omitted
      

      As discussed with Sorabh Hamirwasia the bug is that the subsequent left incoming batch has a schema change when the output batch is having 0 elements because of filter.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shamirwasia Sorabh Hamirwasia
                Reporter:
                shamirwasia Sorabh Hamirwasia
                Reviewer:
                Parth Chandra
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: