Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-989

Cleanup of child blocks after parent execution block is complete

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: Worker
    • Labels:
      None

      Description

      Child execution block working directory is not deleted until the query is complete. there are too many files in directory. we should delete the child execution block

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/103

        TAJO-989: Cleanup of child blocks after parent execution block is comple...

        ...te

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-989

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/103.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #103


        commit 5b55e875e3a8b060dd08c9d3a981062fd61dfda4
        Author: jinossy <jinossy@gmail.com>
        Date: 2014-08-03T14:07:58Z

        TAJO-989: Cleanup of child blocks after parent execution block is complete


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/103 TAJO-989 : Cleanup of child blocks after parent execution block is comple... ...te You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-989 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/103.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #103 commit 5b55e875e3a8b060dd08c9d3a981062fd61dfda4 Author: jinossy <jinossy@gmail.com> Date: 2014-08-03T14:07:58Z TAJO-989 : Cleanup of child blocks after parent execution block is complete
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/103#discussion_r15741058

        — Diff: tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java —
        @@ -363,7 +364,7 @@ public void addTask(QueryUnit task) {

        • It finalizes this subquery. It is only invoked when the subquery is succeeded.
          */
          public void complete() {
        • cleanup();
          + cleanup(getId());
            • End diff –

        You can get the block id by calling ```block.getId()```. So, cleanup() doesn't need to take an execution block id.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/103#discussion_r15741058 — Diff: tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java — @@ -363,7 +364,7 @@ public void addTask(QueryUnit task) { It finalizes this subquery. It is only invoked when the subquery is succeeded. */ public void complete() { cleanup(); + cleanup(getId()); End diff – You can get the block id by calling ```block.getId()```. So, cleanup() doesn't need to take an execution block id.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/103#discussion_r15741073

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/Task.java —
        @@ -27,6 +27,7 @@
        import org.apache.hadoop.conf.Configuration;
        import org.apache.hadoop.fs.*;
        import org.apache.hadoop.util.ReflectionUtils;
        +import org.apache.tajo.ExecutionBlockId;
        — End diff –

        It's an unused import.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/103#discussion_r15741073 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/Task.java — @@ -27,6 +27,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.*; import org.apache.hadoop.util.ReflectionUtils; +import org.apache.tajo.ExecutionBlockId; — End diff – It's an unused import.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/103#issuecomment-51019328

        +1

        This work will remove unnecessary intermediate data earlier than the current implementation. The motivation is reasonable and the patch looks nice to me.

        I leaved one trivial comment. If you agree with my comment, you can commit it after refactoring SubQuery::cleanup method.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/103#issuecomment-51019328 +1 This work will remove unnecessary intermediate data earlier than the current implementation. The motivation is reasonable and the patch looks nice to me. I leaved one trivial comment. If you agree with my comment, you can commit it after refactoring SubQuery::cleanup method.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/103#issuecomment-51144284

        Thank you for the review. I reflected your comments.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/103#issuecomment-51144284 Thank you for the review. I reflected your comments.
        Hide
        jhkim Jinho Kim added a comment -

        I've committed it

        Show
        jhkim Jinho Kim added a comment - I've committed it
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/103

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/103
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #325 (See https://builds.apache.org/job/Tajo-master-build/325/)
        TAJO-989: Cleanup of child blocks after parent execution block is complete. (jinho) (jinossy: rev 0f3412a74bb3c565df1259b19630bc17e1bc69e0)

        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java
        • tajo-core/src/main/proto/TajoWorkerProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
        • tajo-core/src/test/java/org/apache/tajo/worker/TestDeletionService.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #325 (See https://builds.apache.org/job/Tajo-master-build/325/ ) TAJO-989 : Cleanup of child blocks after parent execution block is complete. (jinho) (jinossy: rev 0f3412a74bb3c565df1259b19630bc17e1bc69e0) tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java CHANGES tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java tajo-core/src/test/java/org/apache/tajo/worker/TestDeletionService.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development