Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-673

Assign proper number of tasks when inserting into partitioned table

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      When inserting into partitioned table, if the number of partitions is smaller than cluster concurrency capacity, a query execution is too slow.

      1. TAJO-673_2.patch
        55 kB
        Jaehwa Jung
      2. TAJO-673_3.patch
        55 kB
        Jaehwa Jung
      3. TAJO-673_4.patch
        52 kB
        Jaehwa Jung
      4. TAJO-673_5.patch
        51 kB
        Jaehwa Jung
      5. TAJO-673_6.patch
        15 kB
        Jaehwa Jung
      6. TAJO-673_7.patch
        24 kB
        Jaehwa Jung
      7. TAJO-673_8.patch
        24 kB
        Jaehwa Jung
      8. TAJO-673_9.patch
        24 kB
        Jaehwa Jung
      9. TAJO-673.patch
        11 kB
        Jaehwa Jung

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #296 (See https://builds.apache.org/job/Tajo-master-build/296/)
          TAJO-673: Assign proper number of tasks when inserting into partitioned table. (jaehwa) (jhjung: rev 48dbfd92c4f424ffef6c9aa03e905882bd42bbec)

          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • CHANGES
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashShuffleFileWriteExec.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #296 (See https://builds.apache.org/job/Tajo-master-build/296/ ) TAJO-673 : Assign proper number of tasks when inserting into partitioned table. (jaehwa) (jhjung: rev 48dbfd92c4f424ffef6c9aa03e905882bd42bbec) tajo-core/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java tajo-core/src/main/java/org/apache/tajo/worker/Task.java CHANGES tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashShuffleFileWriteExec.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java
          Hide
          blrunner Jaehwa Jung added a comment -

          This patch got +1 at RB.
          I've just push it to master branch.

          Show
          blrunner Jaehwa Jung added a comment - This patch got +1 at RB. I've just push it to master branch.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12655579/TAJO-673_9.patch
          against master revision 5e2d6c3.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 218 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/465//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/465//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/465//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/465//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/465//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655579/TAJO-673_9.patch against master revision 5e2d6c3. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 218 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/465//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/465//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/465//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/465//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/465//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          I updated unit test cases.

          Show
          blrunner Jaehwa Jung added a comment - I updated unit test cases.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12655519/TAJO-673_8.patch
          against master revision 5e2d6c3.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/464//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/464//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/464//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655519/TAJO-673_8.patch against master revision 5e2d6c3. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/464//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/464//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/464//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          I updated the patch for Hyunsik's review.

          Show
          blrunner Jaehwa Jung added a comment - I updated the patch for Hyunsik's review.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12654076/TAJO-673_7.patch
          against master revision eae4c13.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 209 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/456//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/456//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/456//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/456//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/456//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654076/TAJO-673_7.patch against master revision eae4c13. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 209 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/456//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/456//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/456//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/456//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/456//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          I updated the patch as follows:

          • Divide fetch uris into the the proper number of tasks by IntermediateData output volume. The output volume is 256MB, but you can set it at tajo configuration file. This property name is tajo.scattered.hash.shuffle.split.volume.
          • Adding shuffle output volume to TajoWorkerProtocol. If task complete, then Task::getTaskCompletionReport will set this property.

          For reference, I tested lots of cases on TPC-H benchmarking cluster, and I found that it ran successfully.

          Show
          blrunner Jaehwa Jung added a comment - I updated the patch as follows: Divide fetch uris into the the proper number of tasks by IntermediateData output volume. The output volume is 256MB, but you can set it at tajo configuration file. This property name is tajo.scattered.hash.shuffle.split.volume. Adding shuffle output volume to TajoWorkerProtocol. If task complete, then Task::getTaskCompletionReport will set this property. For reference, I tested lots of cases on TPC-H benchmarking cluster, and I found that it ran successfully.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12653971/TAJO-673_6.patch
          against master revision eae4c13.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 200 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/455//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/455//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/455//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/455//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12653971/TAJO-673_6.patch against master revision eae4c13. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 200 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/455//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/455//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/455//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/455//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          I updated the patch as follows:

          • Apply IntermediateEntry total size to task size in scattered hash shuffle
          • Remove unnecessary configurations
          • Simplify unit test cases for inserting partitioned table
          Show
          blrunner Jaehwa Jung added a comment - I updated the patch as follows: Apply IntermediateEntry total size to task size in scattered hash shuffle Remove unnecessary configurations Simplify unit test cases for inserting partitioned table
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12653819/TAJO-673_5.patch
          against master revision f5e9999.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 209 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/454//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/454//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/454//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/454//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/454//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12653819/TAJO-673_5.patch against master revision f5e9999. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 209 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/454//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/454//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/454//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/454//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/454//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/22374/

          Show
          blrunner Jaehwa Jung added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/22374/
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12651917/TAJO-673_4.patch
          against master revision 1613cd2.

          -1 patch. The patch command could not apply the patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/451//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651917/TAJO-673_4.patch against master revision 1613cd2. -1 patch. The patch command could not apply the patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/451//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment - - edited

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/22374/

          Show
          blrunner Jaehwa Jung added a comment - - edited Updated the review request against branch master in reviewboard https://reviews.apache.org/r/22374/
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12649853/TAJO-673_3.patch
          against master revision f3092c4.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 208 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/449//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/449//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/449//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/449//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/449//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12649853/TAJO-673_3.patch against master revision f3092c4. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 208 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/449//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/449//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/449//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/449//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/449//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          I modified the patch as following:

          • Renamed new shuffle type to scattered hash shuffle.
          • Set TajoConf:SHUFFLE_TASK_NUM_VOLUME to 512MB

          For reference, I tested this patch on my testing cluster with TPC-H dataset.
          I found a problem which produces many empty Tasks. Luckily, Tasks doesn't effect query result.

          But we need to resolve this problem. I think that we should refactor query result stats for removing empty tasks.

          Show
          blrunner Jaehwa Jung added a comment - I modified the patch as following: Renamed new shuffle type to scattered hash shuffle. Set TajoConf:SHUFFLE_TASK_NUM_VOLUME to 512MB For reference, I tested this patch on my testing cluster with TPC-H dataset. I found a problem which produces many empty Tasks. Luckily, Tasks doesn't effect query result. But we need to resolve this problem. I think that we should refactor query result stats for removing empty tasks.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12649494/TAJO-673_2.patch
          against master revision 500c683.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 208 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/446//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/446//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/446//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/446//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/446//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12649494/TAJO-673_2.patch against master revision 500c683. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 208 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-yarn-pullserver. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/446//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/446//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-yarn-pullserver.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/446//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/446//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/446//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/22374/

          Show
          blrunner Jaehwa Jung added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/22374/
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12648955/TAJO-673.patch
          against master revision 500c683.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 198 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in tajo-common tajo-core:
          org.apache.tajo.engine.query.TestCaseByCases
          org.apache.tajo.client.TestTajoClient
          org.apache.tajo.engine.query.TestTablePartitions
          org.apache.tajo.engine.query.TestSortQuery
          org.apache.tajo.engine.query.TestJoinBroadcast
          org.apache.tajo.engine.function.TestBuiltinFunctions
          org.apache.tajo.engine.query.TestNetTypes
          org.apache.tajo.engine.query.TestCTASQuery
          org.apache.tajo.cli.TestTajoCli
          org.apache.tajo.master.querymaster.TestQueryUnitStatusUpdate

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/445//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/445//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/445//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/445//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648955/TAJO-673.patch against master revision 500c683. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 198 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in tajo-common tajo-core: org.apache.tajo.engine.query.TestCaseByCases org.apache.tajo.client.TestTajoClient org.apache.tajo.engine.query.TestTablePartitions org.apache.tajo.engine.query.TestSortQuery org.apache.tajo.engine.query.TestJoinBroadcast org.apache.tajo.engine.function.TestBuiltinFunctions org.apache.tajo.engine.query.TestNetTypes org.apache.tajo.engine.query.TestCTASQuery org.apache.tajo.cli.TestTajoCli org.apache.tajo.master.querymaster.TestQueryUnitStatusUpdate Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/445//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/445//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/445//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/445//console This message is automatically generated.
          Hide
          blrunner Jaehwa Jung added a comment -

          Created a review request against branch master in reviewboard.
          https://reviews.apache.org/r/22374

          Show
          blrunner Jaehwa Jung added a comment - Created a review request against branch master in reviewboard. https://reviews.apache.org/r/22374

            People

            • Assignee:
              blrunner Jaehwa Jung
              Reporter:
              hjkim Hyoungjun Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development