Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6766

Lateral Unnest query : IllegalStateException - rowId in right batch of lateral is smaller than rowId in left batch being processed

    XMLWordPrintableJSON

    Details

      Description

      The error is coming when one batch of streaming agg is split across multiple output batches. In that case the first output batch is sent downstream and in subsequent next() StreamingAgg discards the unprocessed rows in previous incoming and call's next() to upstream again. At this point lateral is waiting to receive the rows for unprocessed ones from right side and instead it see's a batch with lower rowId from streaming agg resulting in the exception.
      This is also a regression in previous behavior where without EMIT outcome support StreamingAgg was handling this case properly. 

      SELECT ttt.number_segments_in_group from BGFsmall t, LATERAL (select l.o.`element`.geo_segments.list as ot from unnest(t.geo_onds.list) l(o)) tt, LATERAL (select count(*) as number_segments_in_group from unnest(tt.ot) ll(o)) ttt limit 3;
      
      

      Log:-

      [Error Id: cfe63aaa-7115-4fab-b0fb-d5098d3b6350 on drill182:31010]
      
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Unexpected case where rowId 0 in right batch of lateral is smaller than rowId 32768 in left batch being processed
      
       
      
      Fragment 0:0
      
       
      
      [Error Id: cfe63aaa-7115-4fab-b0fb-d5098d3b6350 on drill182:31010]
      
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:360) [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:215) [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:326) [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
      
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
      
              at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
      
      Caused by: java.lang.IllegalStateException: Unexpected case where rowId 0 in right batch of lateral is smaller than rowId 32768 in left batch being processed
      
              at org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkState(Preconditions.java:609) ~[drill-shaded-guava-23.0.jar:23.0]
      
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.crossJoinAndOutputRecords(LateralJoinBatch.java:1024) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.produceOutputBatch(LateralJoinBatch.java:575) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.join.LateralJoinBatch.innerNext(LateralJoinBatch.java:238) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:175) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:122) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:112) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:101) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:175) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:122) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:112) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:101) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:175) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:122) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:112) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:175) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:122) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:112) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:143) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:175) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:103) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:93) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:293) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:280) ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
      
              at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_161]
      
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0-mapr-1707.jar:na]
      
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:280) [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
      
              ... 4 common frames omitted
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shamirwasia Sorabh Hamirwasia
                Reporter:
                kedar.behera Kedar Sankar Behera
                Reviewer:
                Boaz Ben-Zvi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: