Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-672

Wrong progress status when overwrites to partition table

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: Data Shuffle
    • Labels:
      None

      Description

      I ran next query in cli. Progress is over 100% and then down to 97%. But finally finished at 100%.

      create table orders_partition (
      o_orderkey      INT8,
      o_custkey       INT8,
      o_orderstatus   TEXT,
      o_totalprice    FLOAT8,
      o_orderdate     TEXT,
      o_orderpriority TEXT,
      o_clerk TEXT,
      o_shippriority  INT4,
      o_comment       TEXT)
      partition by column (o_year text, o_month text);
      
      insert overwrite into orders_partition 
      select o_orderkey, o_custkey, o_orderstatus, o_totalprice, 
      o_orderdate, o_orderpriority, o_clerk, o_shippriority, o_comment, substr(o_orderdate, 1, 4), substr(o_orderdate , 6, 2) 
      from orders;
      
      Progress: 95%, response time: 73.572 sec
      Progress: 95%, response time: 74.573 sec
      Progress: 95%, response time: 75.574 sec
      Progress: 96%, response time: 76.576 sec
      Progress: 96%, response time: 77.578 sec
      Progress: 96%, response time: 78.58 sec
      Progress: 100%, response time: 79.581 sec
      Progress: 100%, response time: 80.582 sec
      Progress: 101%, response time: 81.584 sec
      Progress: 101%, response time: 82.586 sec
      Progress: 101%, response time: 83.587 sec
      Progress: 101%, response time: 84.589 sec
      Progress: 101%, response time: 85.591 sec
      Progress: 101%, response time: 86.593 sec
      Progress: 102%, response time: 87.595 sec
      Progress: 102%, response time: 88.596 sec
      Progress: 102%, response time: 89.598 sec
      Progress: 102%, response time: 90.601 sec
      Progress: 102%, response time: 91.603 sec
      Progress: 102%, response time: 92.605 sec
      Progress: 102%, response time: 93.607 sec
      Progress: 97%, response time: 94.608 sec
      Progress: 97%, response time: 95.61 sec
      
      1. TAJO-672.patch
        2 kB
        Hyoungjun Kim

        Activity

        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Tajo-master-build #111 (See https://builds.apache.org/job/Tajo-master-build/111/)
        TAJO-672: Wrong progress status when overwrites to partition table. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=1fb837e36487227b159129ac724ac8bc13ab1f55)

        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestFetcher.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-build #111 (See https://builds.apache.org/job/Tajo-master-build/111/ ) TAJO-672 : Wrong progress status when overwrites to partition table. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=1fb837e36487227b159129ac724ac8bc13ab1f55 ) CHANGES.txt tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.8.0-build #17 (See https://builds.apache.org/job/Tajo-0.8.0-build/17/)
        TAJO-672: Wrong progress status when overwrites to partition table. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=8cf31be82a87e4a0ca828e363487fc070f83086c)

        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestFetcher.java
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #17 (See https://builds.apache.org/job/Tajo-0.8.0-build/17/ ) TAJO-672 : Wrong progress status when overwrites to partition table. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=8cf31be82a87e4a0ca828e363487fc070f83086c ) tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestFetcher.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
        Hide
        hyunsik Hyunsik Choi added a comment -

        committed it to master branch and 0.8.0.

        Show
        hyunsik Hyunsik Choi added a comment - committed it to master branch and 0.8.0.
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1

        Thank you for quick bug fix. Essentially, this bug is caused by wrong computation in shuffle phase. Since writing partitioned tables take long time, this bug may be produced. Thanks again. I just verified mvn clean install. I'll commit it.

        Show
        hyunsik Hyunsik Choi added a comment - +1 Thank you for quick bug fix. Essentially, this bug is caused by wrong computation in shuffle phase. Since writing partitioned tables take long time, this bug may be produced. Thanks again. I just verified mvn clean install. I'll commit it.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12633313/TAJO-672.patch
        against master revision e3da0ca.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/202//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/202//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/202//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633313/TAJO-672.patch against master revision e3da0ca. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/202//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/202//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/202//console This message is automatically generated.
        Hide
        hjkim Hyoungjun Kim added a comment -

        Progress is incorrectly calculated. I fixed it.
        Please review this patch.

        Show
        hjkim Hyoungjun Kim added a comment - Progress is incorrectly calculated. I fixed it. Please review this patch.

          People

          • Assignee:
            hjkim Hyoungjun Kim
            Reporter:
            hjkim Hyoungjun Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development