Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: SQL Shell
    • Labels:
      None

      Description

      When I run Tajo on two different clusters, the cli prints wrong response time.
      Here is examples.

      • Cluster1: the response time decreases when the query is finished.
        tpch100> select count(*) from customer;
        Progress: 30%, response time: 1.834 sec
        Progress: 30%, response time: 1.834 sec
        Progress: 30%, response time: 2.235 sec
        Progress: 30%, response time: 3.036 sec
        Progress: 43%, response time: 4.037 sec
        Progress: 43%, response time: 5.038 sec
        Progress: 100%, response time: 4.878 sec
        ?count
        -------------------------------
        15000000
        (1 rows, 4.878 sec, 9 B selected)
        
      • Cluster2: the response time jumps when the query is finished.
        tpch_hdfs> select count(*) from customer;
        Progress: 0%, response time: 0.908 sec
        Progress: 0%, response time: 0.909 sec
        Progress: 0%, response time: 1.31 sec
        Progress: 0%, response time: 2.111 sec
        Progress: 48%, response time: 3.111 sec
        Progress: 100%, response time: 24.104 sec
        ?count
        -------------------------------
        15000000
        (1 rows, 24.104 sec, 9 B selected)
        

      I suspect that the wrong information is stored when the query history is written.

        Activity

        Hide
        jihoonson Jihoon Son added a comment -

        The problem is, the start time of a query is set by the Tajo master, but the finish time is set by the Query master.
        So, if the time is not synchronized exactly, this problem can occur.

        To solve this problem, I suggest that the Query master sets the start time as well as the finish time.
        What do you think about it?

        Show
        jihoonson Jihoon Son added a comment - The problem is, the start time of a query is set by the Tajo master, but the finish time is set by the Query master. So, if the time is not synchronized exactly, this problem can occur. To solve this problem, I suggest that the Query master sets the start time as well as the finish time. What do you think about it?
        Hide
        jihoonson Jihoon Son added a comment -

        Ok.
        Allowing the Tajo master to set the start time and the finish time will be the better solution because users think that a query starts when they submit the query, and finishes when they get the results.

        Show
        jihoonson Jihoon Son added a comment - Ok. Allowing the Tajo master to set the start time and the finish time will be the better solution because users think that a query starts when they submit the query, and finishes when they get the results.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jihoonson opened a pull request:

        https://github.com/apache/tajo/pull/369

        TAJO-1321: Cli prints wrong response time.

        As I commented on https://issues.apache.org/jira/browse/TAJO-1321, I made the Tajo master to set the start time and the finish time.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jihoonson/tajo-2 TAJO-1321

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/369.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #369


        commit 109cd30f07d21fe11d6fe96c8ca4ff411706ad27
        Author: Jihoon Son <jihoonson@apache.org>
        Date: 2015-02-01T07:30:57Z

        TAJO-1321


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/369 TAJO-1321 : Cli prints wrong response time. As I commented on https://issues.apache.org/jira/browse/TAJO-1321 , I made the Tajo master to set the start time and the finish time. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1321 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/369.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #369 commit 109cd30f07d21fe11d6fe96c8ca4ff411706ad27 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-02-01T07:30:57Z TAJO-1321
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/369#discussion_r23907071

        — Diff: tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java —
        @@ -428,9 +426,7 @@ public GetQueryListResponse getFinishedQueryList(RpcController controller, GetQu
        infoBuilder.setState(queryInfo.getQueryState());
        infoBuilder.setQuery(queryInfo.getSql());
        infoBuilder.setStartTime(queryInfo.getStartTime());

        • long endTime = (queryInfo.getFinishTime() == 0) ?
        • System.currentTimeMillis() : queryInfo.getFinishTime();
        • infoBuilder.setFinishTime(endTime);
          + infoBuilder.setFinishTime(System.currentTimeMillis());
            • End diff –

        It seems like last request time..
        I suggest that the isFinishState branch in QueryInProgress.hearbeart sets the finish time

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/369#discussion_r23907071 — Diff: tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java — @@ -428,9 +426,7 @@ public GetQueryListResponse getFinishedQueryList(RpcController controller, GetQu infoBuilder.setState(queryInfo.getQueryState()); infoBuilder.setQuery(queryInfo.getSql()); infoBuilder.setStartTime(queryInfo.getStartTime()); long endTime = (queryInfo.getFinishTime() == 0) ? System.currentTimeMillis() : queryInfo.getFinishTime(); infoBuilder.setFinishTime(endTime); + infoBuilder.setFinishTime(System.currentTimeMillis()); End diff – It seems like last request time.. I suggest that the isFinishState branch in QueryInProgress.hearbeart sets the finish time
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/369#issuecomment-72421131

        @jinossy thanks for your review.
        I fixed it.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/369#issuecomment-72421131 @jinossy thanks for your review. I fixed it.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12695890/TAJO-1321.patch
        against master revision release-0.9.0-rc0-161-g1e00759.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 159 new Findbugs (version 2.0.3) warnings.

        -1 release audit. The applied patch generated 882 release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/581//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/581//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/581//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/581//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695890/TAJO-1321.patch against master revision release-0.9.0-rc0-161-g1e00759. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 159 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 882 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/581//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/581//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/581//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/581//console This message is automatically generated.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/369#issuecomment-72431899

        +1 LGTM!
        I've verify on cluster.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/369#issuecomment-72431899 +1 LGTM! I've verify on cluster.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/369

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/369
        Hide
        jihoonson Jihoon Son added a comment -

        Thanks for your review!

        Show
        jihoonson Jihoon Son added a comment - Thanks for your review!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-build #573 (See https://builds.apache.org/job/Tajo-master-build/573/)
        TAJO-1321: Cli prints wrong response time. (jihoon) (jihoonson: rev 58bbb1bb424288fdb3e731374de9938911d8e0f3)

        • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java
        • tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java
        • tajo-core/src/main/proto/QueryCoordinatorProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #573 (See https://builds.apache.org/job/Tajo-master-build/573/ ) TAJO-1321 : Cli prints wrong response time. (jihoon) (jihoonson: rev 58bbb1bb424288fdb3e731374de9938911d8e0f3) tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java CHANGES tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java tajo-core/src/main/proto/QueryCoordinatorProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #212 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/212/)
        TAJO-1321: Cli prints wrong response time. (jihoon) (jihoonson: rev 58bbb1bb424288fdb3e731374de9938911d8e0f3)

        • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
        • tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java
        • CHANGES
        • tajo-core/src/main/proto/QueryCoordinatorProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #212 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/212/ ) TAJO-1321 : Cli prints wrong response time. (jihoon) (jihoonson: rev 58bbb1bb424288fdb3e731374de9938911d8e0f3) tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java CHANGES tajo-core/src/main/proto/QueryCoordinatorProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java

          People

          • Assignee:
            jihoonson Jihoon Son
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development