Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-620

A join query can cause IndexOutOfBoundsException if one of tables is empty.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: distributed query plan
    • Labels:
      None

      Description

      A join query can cause IndexOutOfBoundsException if one of tables is empty. The reproduction condition is very straightforward. We should fix it.

      1. TAJO-620_2.patch
        6 kB
        Jaehwa Jung
      2. TAJO-620.patch
        9 kB
        Jaehwa Jung

        Issue Links

          Activity

          Hide
          hyunsik Hyunsik Choi added a comment -

          I missed the stack trace.

          java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
                  at java.util.ArrayList.rangeCheck(ArrayList.java:635)
                  at java.util.ArrayList.get(ArrayList.java:411)
                  at org.apache.tajo.master.querymaster.Repartitioner.scheduleFragmentsForJoinQuery(Repartitioner.java:103)
                  at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.schedule(SubQuery.java:730)
                  at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:579)
                  at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:560)
                  at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
                  at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
                  at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
                  at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
                  at org.apache.tajo.master.querymaster.SubQuery.handle(SubQuery.java:537)
                  at org.apache.tajo.master.querymaster.Query$StartTransition.transition(Query.java:332)
                  at org.apache.tajo.master.querymaster.Query$StartTransition.transition(Query.java:321)
                  at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
                  at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
                  at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
                  at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
                  at org.apache.tajo.master.querymaster.Query.handle(Query.java:646)
                  at org.apache.tajo.master.querymaster.Query.handle(Query.java:58)
                  at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137)
                  at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79)
                  at java.lang.Thread.run(Thread.java:744)
          
          Show
          hyunsik Hyunsik Choi added a comment - I missed the stack trace. java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck(ArrayList.java:635) at java.util.ArrayList.get(ArrayList.java:411) at org.apache.tajo.master.querymaster.Repartitioner.scheduleFragmentsForJoinQuery(Repartitioner.java:103) at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.schedule(SubQuery.java:730) at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:579) at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:560) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.tajo.master.querymaster.SubQuery.handle(SubQuery.java:537) at org.apache.tajo.master.querymaster.Query$StartTransition.transition(Query.java:332) at org.apache.tajo.master.querymaster.Query$StartTransition.transition(Query.java:321) at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.tajo.master.querymaster.Query.handle(Query.java:646) at org.apache.tajo.master.querymaster.Query.handle(Query.java:58) at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137) at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79) at java.lang.Thread.run(Thread.java:744)
          Hide
          blrunner Jaehwa Jung added a comment -

          I uploaded the patch, and I created a new RB https://reviews.apache.org/r/18556/.

          Show
          blrunner Jaehwa Jung added a comment - I uploaded the patch, and I created a new RB https://reviews.apache.org/r/18556/ .
          Hide
          blrunner Jaehwa Jung added a comment -

          I have been trying to resolve this issue and TAJO-427. And I found that we have to handle this issue differently for join type.

          PhysicalPlanner creates PhysicalExec assuming that Fragment has data. But in outer join case, we enable PhysicalExec to handle empty Fragment. If we adopt this case, we should find better solutions.

          So, I wish to resolve just inner join case at this issue, and I wish to resolve outer join case at TAJO-427.

          Show
          blrunner Jaehwa Jung added a comment - I have been trying to resolve this issue and TAJO-427 . And I found that we have to handle this issue differently for join type. PhysicalPlanner creates PhysicalExec assuming that Fragment has data. But in outer join case, we enable PhysicalExec to handle empty Fragment. If we adopt this case, we should find better solutions. So, I wish to resolve just inner join case at this issue, and I wish to resolve outer join case at TAJO-427 .
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631443/TAJO-620.patch
          against master revision 56fbd99.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 178 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/165//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/165//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/165//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631443/TAJO-620.patch against master revision 56fbd99. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 178 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/165//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/165//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/165//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          I uploaded the second patch, and I also updated the RB.

          Show
          blrunner Jaehwa Jung added a comment - I uploaded the second patch, and I also updated the RB.
          Hide
          blrunner Jaehwa Jung added a comment -

          This issue go +1 at RB.
          I committed it to the master branch and branch-0.8.0.

          Show
          blrunner Jaehwa Jung added a comment - This issue go +1 at RB. I committed it to the master branch and branch-0.8.0.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12631450/TAJO-620_2.patch
          against master revision 56fbd99.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 178 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/167//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/167//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/167//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12631450/TAJO-620_2.patch against master revision 56fbd99. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 178 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/167//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/167//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/167//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #89 (See https://builds.apache.org/job/Tajo-master-build/89/)
          TAJO-620: A join query can cause IndexOutOfBoundsException if one of tables is empty. (jaehwa) (jhjung: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=db5c017d416561c9dac08c093558d7344aa53be1)

          • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testInnerJoinWithEmptyTable.sql
          • CHANGES.txt
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
          • tajo-core/tajo-core-backend/src/test/resources/results/TestJoinQuery/testInnerJoinWithEmptyTable.result
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #89 (See https://builds.apache.org/job/Tajo-master-build/89/ ) TAJO-620 : A join query can cause IndexOutOfBoundsException if one of tables is empty. (jaehwa) (jhjung: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=db5c017d416561c9dac08c093558d7344aa53be1 ) tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testInnerJoinWithEmptyTable.sql CHANGES.txt tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java tajo-core/tajo-core-backend/src/test/resources/results/TestJoinQuery/testInnerJoinWithEmptyTable.result

            People

            • Assignee:
              blrunner Jaehwa Jung
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development