[DRILL-6137] Join Failure When Some Json File Partitions Empty

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.13.0
• Assignee: Timothy Farkas
• Reporter: Timothy Farkas
• Created: 05/Feb/18 20:38
• Updated: 01/Mar/18 22:32
• Resolved: 01/Mar/18 22:32

Description

The following exception can occurr when the following query is executed:

select t.p_partkey, t1.ps_suppkey from dfs.`join/empty_part/part` as t RIGHT JOIN dfs.`join/empty_part/partsupp` as t1 ON t.p_partkey = t1.ps_partkey where t1.ps_partkey > 1

• part has one nonempty file 0_0_0.json
• partsupp has one nonempty file 0_0_0.json and one empty file 0_0_1.json

  (java.lang.IllegalStateException) next() [on #10, RemovingRecordBatch] called again after it returned NONE.  Caller should not have called next() again.
    org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():220
    org.apache.drill.exec.record.AbstractRecordBatch.next():119
    org.apache.drill.exec.test.generated.HashJoinProbeGen2.executeProbePhase():119
    org.apache.drill.exec.test.generated.HashJoinProbeGen2.probeAndProject():227
    org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():222
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():228
    org.apache.drill.exec.record.AbstractRecordBatch.next():119
    org.apache.drill.exec.record.AbstractRecordBatch.next():109
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():228
    org.apache.drill.exec.record.AbstractRecordBatch.next():119
    org.apache.drill.exec.record.AbstractRecordBatch.next():109
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():228
    org.apache.drill.exec.physical.impl.BaseRootExec.next():105
    org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
    org.apache.drill.exec.physical.impl.BaseRootExec.next():95
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():233
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():415
    org.apache.hadoop.security.UserGroupInformation.doAs():1657
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745