Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1802

PythonScriptEngine copies controller and tajo util whenever it is initialized

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Function/UDF
    • Labels:
      None

      Description

      In the current implementation, python script files are copied and deleted whenever a task is launched and stopped. This incurs two problems.

      • Frequent copy of script files degrades the overall performance.
      • Among python scripts, tajo_util.py is shared by multiple tasks of a worker. This incurs I/O problem during unit tests because the shared file is also removed when a task is stopped.
      1. TAJO-1802_2.patch
        18 kB
        Jihoon Son
      2. TAJO-1802.patch
        17 kB
        Jihoon Son

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.11.0-build #6 (See https://builds.apache.org/job/Tajo-0.11.0-build/6/)
        TAJO-1802: PythonScriptEngine copies controller and tajo util whenever it is initialized. (jihoonson: rev d214d50c68b895fa0cba4e750ab942f3245b36a9)

        • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
        • CHANGES
        • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
        • tajo-core/src/main/resources/python/controller.py
        • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java
        • tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.0-build #6 (See https://builds.apache.org/job/Tajo-0.11.0-build/6/ ) TAJO-1802 : PythonScriptEngine copies controller and tajo util whenever it is initialized. (jihoonson: rev d214d50c68b895fa0cba4e750ab942f3245b36a9) tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java CHANGES tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/src/main/resources/python/controller.py tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #825 (See https://builds.apache.org/job/Tajo-master-build/825/)
        TAJO-1802: PythonScriptEngine copies controller and tajo util whenever it is initialized. (jihoonson: rev ace0480feacab917e6c73dd37314994050ab6207)

        • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java
        • tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java
        • CHANGES
        • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
        • tajo-core/src/main/resources/python/controller.py
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #825 (See https://builds.apache.org/job/Tajo-master-build/825/ ) TAJO-1802 : PythonScriptEngine copies controller and tajo util whenever it is initialized. (jihoonson: rev ace0480feacab917e6c73dd37314994050ab6207) tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java CHANGES tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/src/main/resources/python/controller.py
        Hide
        jihoonson Jihoon Son added a comment -

        Thanks for quick review!
        Committed to master and 0.11.

        Show
        jihoonson Jihoon Son added a comment - Thanks for quick review! Committed to master and 0.11.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #469 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/469/)
        TAJO-1802: PythonScriptEngine copies controller and tajo util whenever it is initialized. (jihoonson: rev ace0480feacab917e6c73dd37314994050ab6207)

        • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
        • tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java
        • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java
        • CHANGES
        • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java
        • tajo-core/src/main/resources/python/controller.py
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #469 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/469/ ) TAJO-1802 : PythonScriptEngine copies controller and tajo util whenever it is initialized. (jihoonson: rev ace0480feacab917e6c73dd37314994050ab6207) tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java CHANGES tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java tajo-core/src/main/resources/python/controller.py tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/715

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/715
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/715#issuecomment-136973837

        +1 LGTM!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/715#issuecomment-136973837 +1 LGTM!
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12753689/TAJO-1802_2.patch
        against master revision release-0.9.0-rc0-434-g2c9305a.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

        -1 release audit. The applied patch generated 1 release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core tajo-plan.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/835//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/835//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
        Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/835//findbugsResult
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/835//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12753689/TAJO-1802_2.patch against master revision release-0.9.0-rc0-434-g2c9305a. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. -1 release audit. The applied patch generated 1 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core tajo-plan. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/835//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/835//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/835//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/835//console This message is automatically generated.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/715#issuecomment-136937052

        Thank you for the nice comment. I'll change it.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/715#issuecomment-136937052 Thank you for the nice comment. I'll change it.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/715#issuecomment-136936987

        In my opinion, once copied controller.py and tajo_util.py, It does not need to overwrite.
        Can you move to TajoWorker.serviceInit and TajoMaster.serviceInit ?

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/715#issuecomment-136936987 In my opinion, once copied controller.py and tajo_util.py, It does not need to overwrite. Can you move to TajoWorker.serviceInit and TajoMaster.serviceInit ?
        Hide
        jihoonson Jihoon Son added a comment -

        This patch is ready for review.

        Show
        jihoonson Jihoon Son added a comment - This patch is ready for review.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jihoonson opened a pull request:

        https://github.com/apache/tajo/pull/715

        TAJO-1802: PythonScriptEngine copies controller and tajo util whenever it is initialized

        I've changed to copy python files exactly once before script engine is started in master. In worker, python files are treated as one of shared resources.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jihoonson/tajo-2 TAJO-1802

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/715.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #715


        commit 37eabf2f274f8d09b34e52e511e2f1844d6d8233
        Author: Jihoon Son <jihoonson@apache.org>
        Date: 2015-08-31T09:51:40Z

        TAJO-1802

        commit d335fd25f2751a210fbc25ebf4360cacea0ff1f2
        Author: Jihoon Son <jihoonson@apache.org>
        Date: 2015-08-31T11:06:37Z

        TAJO-1802


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/715 TAJO-1802 : PythonScriptEngine copies controller and tajo util whenever it is initialized I've changed to copy python files exactly once before script engine is started in master. In worker, python files are treated as one of shared resources. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1802 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/715.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #715 commit 37eabf2f274f8d09b34e52e511e2f1844d6d8233 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-08-31T09:51:40Z TAJO-1802 commit d335fd25f2751a210fbc25ebf4360cacea0ff1f2 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-08-31T11:06:37Z TAJO-1802

          People

          • Assignee:
            jihoonson Jihoon Son
            Reporter:
            jihoonson Jihoon Son
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development