Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: QueryMaster
    • Labels:
      None

      Description

      In the current version, killQuery feature is partially implemented. We need to complete this feature and add a command to tsql.

      1. TAJO-305_140217_170533.patch
        120 kB
        Hyunsik Choi
      2. TAJO-305_140217_165738.patch
        120 kB
        Hyunsik Choi
      3. TAJO-305_140217_153905.patch
        120 kB
        Hyunsik Choi
      4. TAJO-305_20140216_01:54:24.patch
        120 kB
        Hyunsik Choi
      5. TAJO-305.patch
        102 kB
        Hyunsik Choi

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #66 (See https://builds.apache.org/job/Tajo-master-build/66/)
          TAJO-305: Implement killQuery feature. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=ae541ffae406faa53ace5ad19cb503b96c39c549)

          • tajo-client/src/main/java/org/apache/tajo/client/TajoClient.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/LocalTaskEventType.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SortBasedColPartitionStoreExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQueryEventType.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/function/builtin/Sleep.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryFinishEvent.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQuerySucceeEvent.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/TaskSchedulerEvent.java
          • tajo-core/tajo-core-backend/src/main/resources/webapps/worker/querydetail.jsp
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
          • tajo-core/tajo-core-backend/src/main/proto/QueryMasterProtocol.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryMasterQueryCompletedEvent.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQueryCompletedEvent.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/TaskAttemptEventType.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQueryState.java
          • CHANGES.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQueryTaskEvent.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestTajoClient.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
          • tajo-common/src/main/proto/tajo_protos.proto
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryEventType.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/AbstractTaskScheduler.java
          • tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/tajo-core-backend/src/main/resources/webapps/admin/query.jsp
          • tajo-client/src/main/java/org/apache/tajo/client/TajoAdmin.java
          • tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryJobEvent.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryCompletedEvent.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #66 (See https://builds.apache.org/job/Tajo-master-build/66/ ) TAJO-305 : Implement killQuery feature. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=ae541ffae406faa53ace5ad19cb503b96c39c549 ) tajo-client/src/main/java/org/apache/tajo/client/TajoClient.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/LocalTaskEventType.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SortBasedColPartitionStoreExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQueryEventType.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/function/builtin/Sleep.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryFinishEvent.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQuerySucceeEvent.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/TaskSchedulerEvent.java tajo-core/tajo-core-backend/src/main/resources/webapps/worker/querydetail.jsp tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java tajo-core/tajo-core-backend/src/main/proto/QueryMasterProtocol.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryMasterQueryCompletedEvent.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQueryCompletedEvent.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/TaskAttemptEventType.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQueryState.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/SubQueryTaskEvent.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/client/TestTajoClient.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoContainerProxy.java tajo-common/src/main/proto/tajo_protos.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryEventType.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/AbstractTaskScheduler.java tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto tajo-core/tajo-core-backend/src/main/resources/webapps/admin/query.jsp tajo-client/src/main/java/org/apache/tajo/client/TajoAdmin.java tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryJobEvent.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryCompletedEvent.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java
          Hide
          hyunsik Hyunsik Choi added a comment -

          This issue got +1 on RB. I missed this mention.

          Show
          hyunsik Hyunsik Choi added a comment - This issue got +1 on RB. I missed this mention.
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed it to master branch. Thanks for the reviews.

          Show
          hyunsik Hyunsik Choi added a comment - committed it to master branch. Thanks for the reviews.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12629330/TAJO-305_140217_170533.patch
          against master revision e2a7dff.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 175 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-client tajo-common tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/133//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/133//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/133//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629330/TAJO-305_140217_170533.patch against master revision e2a7dff. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 175 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-client tajo-common tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/133//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/133//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/133//console This message is automatically generated.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12629329/TAJO-305_140217_165738.patch
          against master revision e2a7dff.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 175 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-client tajo-common tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/132//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/132//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/132//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629329/TAJO-305_140217_165738.patch against master revision e2a7dff. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 175 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-client tajo-common tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/132//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/132//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/132//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          reflected Jiho's comment.

          Updated the review request against branch master in reviewboard

          Show
          hyunsik Hyunsik Choi added a comment - reflected Jiho's comment. Updated the review request against branch master in reviewboard
          Hide
          hyunsik Hyunsik Choi added a comment -

          uploaded the latest patch

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/18164/

          Show
          hyunsik Hyunsik Choi added a comment - uploaded the latest patch Updated the review request against branch master in reviewboard https://reviews.apache.org/r/18164/
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12629317/TAJO-305_140217_153905.patch
          against master revision e2a7dff.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 175 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-client tajo-common tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/131//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/131//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/131//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629317/TAJO-305_140217_153905.patch against master revision e2a7dff. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 175 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-client tajo-common tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/131//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/131//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/131//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          I've just uploaded the rebased patch. Thanks.

          Show
          hyunsik Hyunsik Choi added a comment - I've just uploaded the rebased patch. Thanks.
          Hide
          blrunner Jaehwa Jung added a comment -

          Hi, Hyunsik Choi.

          I couldn't apply your patch because of source conflicts. Could you sync by apache latest version?

          Show
          blrunner Jaehwa Jung added a comment - Hi, Hyunsik Choi . I couldn't apply your patch because of source conflicts. Could you sync by apache latest version?
          Hide
          hyunsik Hyunsik Choi added a comment -

          reflected the comments on RB.

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/18164/

          Show
          hyunsik Hyunsik Choi added a comment - reflected the comments on RB. Updated the review request against branch master in reviewboard https://reviews.apache.org/r/18164/
          Hide
          hyunsik Hyunsik Choi added a comment -

          I've just submitted the patch. You can kill a running query as follows:

          hyunsik@Hyunsiks-MacBook-Pro:tajo-0.8.0-SNAPSHOT$ bin/tajo admin -list
          QueryId              State   StartTime            Query                         
          -------------------- ------- -------------------- ------------------------------
          q_1392482437956_0001 RUNNING 2014-02-16 01:56:59  select * from lineitem orde...
          hyunsik@Hyunsiks-MacBook-Pro:tajo-0.8.0-SNAPSHOT$ bin/tajo admin -kill q_1392482437956_0001 
          q_1392482437956_0001 is killed.
          hyunsik@Hyunsiks-MacBook-Pro:tajo-0.8.0-SNAPSHOT$ 
          

          This work requires the changes of state machines of Query, SubQuery, QueryUnit and QueryUnitAttempt. This patch modified them, and additionally it makes them more robust by adding some transitions. Especially, a query gracefully kills running and failed tasks when the query is failed.

          Show
          hyunsik Hyunsik Choi added a comment - I've just submitted the patch. You can kill a running query as follows: hyunsik@Hyunsiks-MacBook-Pro:tajo-0.8.0-SNAPSHOT$ bin/tajo admin -list QueryId State StartTime Query -------------------- ------- -------------------- ------------------------------ q_1392482437956_0001 RUNNING 2014-02-16 01:56:59 select * from lineitem orde... hyunsik@Hyunsiks-MacBook-Pro:tajo-0.8.0-SNAPSHOT$ bin/tajo admin -kill q_1392482437956_0001 q_1392482437956_0001 is killed. hyunsik@Hyunsiks-MacBook-Pro:tajo-0.8.0-SNAPSHOT$ This work requires the changes of state machines of Query, SubQuery, QueryUnit and QueryUnitAttempt. This patch modified them, and additionally it makes them more robust by adding some transitions. Especially, a query gracefully kills running and failed tasks when the query is failed.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/18164/

          Show
          hyunsik Hyunsik Choi added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/18164/
          Hide
          hyunsik Hyunsik Choi added a comment -

          Created a review request against branch master in reviewboard
          https://reviews.apache.org/r/18164/

          Show
          hyunsik Hyunsik Choi added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/18164/
          Hide
          hyunsik Hyunsik Choi added a comment -

          Thank you for your understanding. Actually, I would like to include this feature to 0.8 release. I'll take it

          Show
          hyunsik Hyunsik Choi added a comment - Thank you for your understanding. Actually, I would like to include this feature to 0.8 release. I'll take it
          Hide
          coderplay Min Zhou added a comment - - edited

          That's ok. More likely I have no time porting my patch to trunk. If you like to finish this ticket, please do it

          Show
          coderplay Min Zhou added a comment - - edited That's ok. More likely I have no time porting my patch to trunk. If you like to finish this ticket, please do it
          Hide
          hyunsik Hyunsik Choi added a comment -

          That's fine. Please go ahead

          Show
          hyunsik Hyunsik Choi added a comment - That's fine. Please go ahead
          Hide
          coderplay Min Zhou added a comment -

          oops, Sorry for the late. I already implemented it , but the trunk move fast. Please go ahead.

          Show
          coderplay Min Zhou added a comment - oops, Sorry for the late. I already implemented it , but the trunk move fast. Please go ahead.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Min,

          You seems busy because you started memory cached table (TAJO-472). If you are Ok, could I take this issue?

          Show
          hyunsik Hyunsik Choi added a comment - Min, You seems busy because you started memory cached table ( TAJO-472 ). If you are Ok, could I take this issue?
          Hide
          hyunsik Hyunsik Choi added a comment -

          Actually, the code was contributed by another guy, and I reviewed roughly that patch. So, that's my fault. Ideally, closeQuery and killQuery should be placed in TajoMaster. When TajoMaster receives the killQuery or closeQuery, TajoMaster handles them and should forward them to QueryMaster.

          Happy thanksgiving!

          Show
          hyunsik Hyunsik Choi added a comment - Actually, the code was contributed by another guy, and I reviewed roughly that patch. So, that's my fault. Ideally, closeQuery and killQuery should be placed in TajoMaster. When TajoMaster receives the killQuery or closeQuery, TajoMaster handles them and should forward them to QueryMaster. Happy thanksgiving!
          Hide
          coderplay Min Zhou added a comment - - edited

          Hi Hyunsik,

          Never mind that. TajoClient has 2 methods with one named killQuery and the other named closeQuery. The former calls TajoMaster while the latter calls QueryMaster. I still don't know which way do you decide to choose? Killing a query from client directly through QueryMaster?

          BTW, happy thanksgiving

          Min

          Show
          coderplay Min Zhou added a comment - - edited Hi Hyunsik, Never mind that. TajoClient has 2 methods with one named killQuery and the other named closeQuery. The former calls TajoMaster while the latter calls QueryMaster. I still don't know which way do you decide to choose? Killing a query from client directly through QueryMaster? BTW, happy thanksgiving Min
          Hide
          hyunsik Hyunsik Choi added a comment -

          Min,

          I'm very sorry for my laziness of refactoring. You read that code exactly. killQuery in TajoClient has wrong name. That should be closeQuery exactly. In addition, stopQuery() in QueryMaster also should be closeQuery(). In TajoClient, killQuery implementation should be rewritten completely. I also misunderstood the progress of implementation.

          If you find such wrong names, feel free to make other jira issues or refactor the names when needed while you doing this issue.

          Show
          hyunsik Hyunsik Choi added a comment - Min, I'm very sorry for my laziness of refactoring. You read that code exactly. killQuery in TajoClient has wrong name. That should be closeQuery exactly. In addition, stopQuery() in QueryMaster also should be closeQuery(). In TajoClient, killQuery implementation should be rewritten completely. I also misunderstood the progress of implementation. If you find such wrong names, feel free to make other jira issues or refactor the names when needed while you doing this issue.
          Hide
          coderplay Min Zhou added a comment -

          Read your code more carefully than before. Seems TajoClient.java already has a killQuery method which communicate with TajoMaster rather than a QueryMaster. Meanwhile there is a method named closeQuery which will kill a Query through QueryMaster. A little bit confused.

          Could you tell me which way would you like to choose?

          Show
          coderplay Min Zhou added a comment - Read your code more carefully than before. Seems TajoClient.java already has a killQuery method which communicate with TajoMaster rather than a QueryMaster. Meanwhile there is a method named closeQuery which will kill a Query through QueryMaster. A little bit confused. Could you tell me which way would you like to choose?
          Hide
          coderplay Min Zhou added a comment - - edited

          OMG.. Only recently did I realize the way I tried to kill a Query is quite different from yours. I added a killQuery rpc call in TajoMasterClientProtocol.proto thus client can remotely request master stop the query. Tajomaster will forward the request to QueryMaster and then QueryMaster receives a kill event to drive the state machine of query / subquery/task/taskattempt and finially recycle all the resource this query lanuches.

          You added a rpc call in QueryMasterClientProtocol.proto and more than half of logic for killing a query had done. TajoClient hold a mapping from queryId to its QueryMaster. Just need to retrieve a QueryMaster by queryId, and call that rpc method. Everything should be done.

          My bad, sorry.

          Show
          coderplay Min Zhou added a comment - - edited OMG.. Only recently did I realize the way I tried to kill a Query is quite different from yours. I added a killQuery rpc call in TajoMasterClientProtocol.proto thus client can remotely request master stop the query. Tajomaster will forward the request to QueryMaster and then QueryMaster receives a kill event to drive the state machine of query / subquery/task/taskattempt and finially recycle all the resource this query lanuches. You added a rpc call in QueryMasterClientProtocol.proto and more than half of logic for killing a query had done. TajoClient hold a mapping from queryId to its QueryMaster. Just need to retrieve a QueryMaster by queryId, and call that rpc method. Everything should be done. My bad, sorry.
          Hide
          coderplay Min Zhou added a comment -

          Thank you for you information. Keep watching
          Do assure can finish much earlier than the end of this year

          Show
          coderplay Min Zhou added a comment - Thank you for you information. Keep watching Do assure can finish much earlier than the end of this year
          Hide
          hyunsik Hyunsik Choi added a comment -

          It would be better that this is included in 0.8-incubating release. 0.8-incubating will be released after the core features planed in our 0.8 roadmap (http://wiki.apache.org/tajo/Roadmap) are completed. So, we don't have tight deadline, and we are just expecting that 0.8 will be the end of this year. I hope that you don't feel any duty and just enjoy contribution to Tajo.

          Besides, because you already start this work, I'll start to refactor some parts that would be overlapped with your work.
          https://issues.apache.org/jira/browse/TAJO-323

          Show
          hyunsik Hyunsik Choi added a comment - It would be better that this is included in 0.8-incubating release. 0.8-incubating will be released after the core features planed in our 0.8 roadmap ( http://wiki.apache.org/tajo/Roadmap ) are completed. So, we don't have tight deadline, and we are just expecting that 0.8 will be the end of this year. I hope that you don't feel any duty and just enjoy contribution to Tajo. Besides, because you already start this work, I'll start to refactor some parts that would be overlapped with your work. https://issues.apache.org/jira/browse/TAJO-323
          Hide
          coderplay Min Zhou added a comment -

          Yeah, the same as I thought. I already finished 1/4 code of the whole process of killing a query.
          Contributing tajo is my personal behavior, I always do it off duty. So btw do you have a deadline for this issue? I can make a arrangement to finish this

          Min

          Show
          coderplay Min Zhou added a comment - Yeah, the same as I thought. I already finished 1/4 code of the whole process of killing a query. Contributing tajo is my personal behavior, I always do it off duty. So btw do you have a deadline for this issue? I can make a arrangement to finish this Min
          Hide
          hyunsik Hyunsik Choi added a comment -

          They have wrong names. Before few months ago, we were changing their names into MR-similar names for developers understanding. But, we didn't complete that work. QueryUnit corresponds to Task, and QueryUnitAttempt corresponds to TaskAttempt. We should correct those names.

          Thank you for your report. I'll correct those names in a separate jira.

          Show
          hyunsik Hyunsik Choi added a comment - They have wrong names. Before few months ago, we were changing their names into MR-similar names for developers understanding. But, we didn't complete that work. QueryUnit corresponds to Task, and QueryUnitAttempt corresponds to TaskAttempt. We should correct those names. Thank you for your report. I'll correct those names in a separate jira.
          Hide
          coderplay Min Zhou added a comment -

          Hi Hyunsik,

          I am confused about the terms in tajo. For mapreduce, the hierarchy is job->task->taskattempt
          For tajo, it is Query -> SubQuery -> QueryUnit -> QueryUnitAttempt, right?
          But I saw there is task, taskattempt in the code d

          dispatcher.register(TaskEventType.class, new TaskEventDispatcher());
          dispatcher.register(TaskAttemptEventType.class, new TaskAttemptEventDispatcher());
          

          What does the task and taskattempt mean? A little bit confused

          Min

          Show
          coderplay Min Zhou added a comment - Hi Hyunsik, I am confused about the terms in tajo. For mapreduce, the hierarchy is job->task->taskattempt For tajo, it is Query -> SubQuery -> QueryUnit -> QueryUnitAttempt, right? But I saw there is task, taskattempt in the code d dispatcher.register(TaskEventType.class, new TaskEventDispatcher()); dispatcher.register(TaskAttemptEventType.class, new TaskAttemptEventDispatcher()); What does the task and taskattempt mean? A little bit confused Min
          Hide
          coderplay Min Zhou added a comment -

          @Hyunsik,

          Please take a look at TAJO-320.

          Show
          coderplay Min Zhou added a comment - @Hyunsik, Please take a look at TAJO-320 .
          Hide
          hyunsik Hyunsik Choi added a comment -

          Sounds great! The state machine diagram would be very helpful for all.

          Show
          hyunsik Hyunsik Choi added a comment - Sounds great! The state machine diagram would be very helpful for all.
          Hide
          coderplay Min Zhou added a comment -

          Yeah, I create a jira for visualizing state machines.

          Show
          coderplay Min Zhou added a comment - Yeah, I create a jira for visualizing state machines.
          Hide
          coderplay Min Zhou added a comment -

          Before that I'd like visualize each statemachine for good design of killing a query. Actually, yarn has a tool. I will create another jira to implement it.

          Show
          coderplay Min Zhou added a comment - Before that I'd like visualize each statemachine for good design of killing a query. Actually, yarn has a tool. I will create another jira to implement it.
          Hide
          blrunner Jaehwa Jung added a comment -

          Great!
          Welcome to Tajo, Min.

          Show
          blrunner Jaehwa Jung added a comment - Great! Welcome to Tajo, Min.
          Hide
          hyunsik Hyunsik Choi added a comment -

          I've assigned it to you. If there are necessary works for this issue, feel free to suggest anything. Thank you!

          Show
          hyunsik Hyunsik Choi added a comment - I've assigned it to you. If there are necessary works for this issue, feel free to suggest anything. Thank you!
          Hide
          hyunsik Hyunsik Choi added a comment -

          Thank you for your participation. You can start this issue now. Actually, now Jira doesn't work well. After a while, I'll add you as a contributor and assign this issue to you.

          Show
          hyunsik Hyunsik Choi added a comment - Thank you for your participation. You can start this issue now. Actually, now Jira doesn't work well. After a while, I'll add you as a contributor and assign this issue to you.
          Hide
          coderplay Min Zhou added a comment -

          Hi Hyunsik,

          Thank you for your agreement. I'd like to take this issue, may I?

          Min

          Show
          coderplay Min Zhou added a comment - Hi Hyunsik, Thank you for your agreement. I'd like to take this issue, may I? Min
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1
          I agree with your mention. I like that form.

          Show
          hyunsik Hyunsik Choi added a comment - +1 I agree with your mention. I like that form.
          Hide
          coderplay Min Zhou added a comment -

          It's not convenient if we implement it in tsql, due to that the query will be killed always hung on the tajo's interactive interface, tsql. we would need to start a new interactive session for the purpose. Sometimes tajo queries would be involved into a workflow system, like azkaban/ooize. Interactive interface may not suitable for those system to kill a query. Command should be better. How do you like this form?

          bin/tajo query -kill query_id
          
          Show
          coderplay Min Zhou added a comment - It's not convenient if we implement it in tsql, due to that the query will be killed always hung on the tajo's interactive interface, tsql. we would need to start a new interactive session for the purpose. Sometimes tajo queries would be involved into a workflow system, like azkaban/ooize. Interactive interface may not suitable for those system to kill a query. Command should be better. How do you like this form? bin/tajo query -kill query_id

            People

            • Assignee:
              hyunsik Hyunsik Choi
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development