Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: RPC
    • Labels:
      None

      Description

      In the current implementation, all client rpc use same channel pool.
      It can cause a channel closed exception. we need both shared pool and new pool.
      In details,

      • Fix the TajoAsyncDispatcher hang
      • Fix the Fetcher timeout
      • Fix the TaskRunner thread leak
      • Fix the client rpc reconnecting
      • Fix the unittest failure(No available resources)
      • Improve RPC thread sharing
      1. TAJO-598.patch
        125 kB
        Jinho Kim
      2. TAJO-598_20140214_20:18:12.patch
        120 kB
        Jinho Kim

        Issue Links

          Activity

          Hide
          jhkim Jinho Kim added a comment -

          Created a review request against branch master in reviewboard
          https://reviews.apache.org/r/18009/

          Show
          jhkim Jinho Kim added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/18009/
          Hide
          jhkim Jinho Kim added a comment -

          Unfortunately, I found memory leak caused by query history. but I will fix the bug in new jira.

          Show
          jhkim Jinho Kim added a comment - Unfortunately, I found memory leak caused by query history. but I will fix the bug in new jira.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12628498/TAJO-598.patch
          against master revision 519a16a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 239 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-server tajo-client tajo-common tajo-core/tajo-core-backend tajo-core/tajo-core-pullserver tajo-rpc.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/119//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/119//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/119//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/119//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/119//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628498/TAJO-598.patch against master revision 519a16a. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 239 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-server tajo-client tajo-common tajo-core/tajo-core-backend tajo-core/tajo-core-pullserver tajo-rpc. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/119//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/119//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/119//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/119//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/119//console This message is automatically generated.
          Hide
          jhkim Jinho Kim added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/18009/

          Show
          jhkim Jinho Kim added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/18009/
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12628994/TAJO-598_20140214_20%3A18%3A12.patch
          against master revision 519a16a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 239 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-server tajo-client tajo-common tajo-core/tajo-core-backend tajo-core/tajo-core-pullserver tajo-rpc.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/124//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/124//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/124//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/124//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-pullserver.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/124//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628994/TAJO-598_20140214_20%3A18%3A12.patch against master revision 519a16a. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 239 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-server tajo-client tajo-common tajo-core/tajo-core-backend tajo-core/tajo-core-pullserver tajo-rpc. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/124//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/124//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/124//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/124//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-pullserver.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/124//console This message is automatically generated.
          Hide
          jhkim Jinho Kim added a comment -

          This issue got +1 from reviewboard.
          I've just committed it to master branch.

          Show
          jhkim Jinho Kim added a comment - This issue got +1 from reviewboard. I've just committed it to master branch.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #65 (See https://builds.apache.org/job/Tajo-master-build/65/)
          TAJO-598: Refactoring Tajo RPC. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e2a7dffdb652c77dd35a917134cef15b3d54d274)

          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcServer.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnContainerProxy.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/UnaryPhysicalExec.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BinaryPhysicalExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/metrics/reporter/TajoMetricsScheduledReporter.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Fetcher.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java
          • tajo-rpc/src/test/java/org/apache/tajo/rpc/TestBlockingRpc.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcClient.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestGlobalPlanner.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/RpcConnectionPool.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/RpcChannelFactory.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
          • tajo-client/src/main/java/org/apache/tajo/client/TajoClient.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/StoreTableExec.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BNLJoinExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java
          • CHANGES.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestFetcher.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/metrics/reporter/MetricsFileScheduledReporter.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyServerBase.java
          • tajo-rpc/src/test/java/org/apache/tajo/rpc/TestAsyncRpc.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcClient.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnTaskRunnerLauncherImpl.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/SimpleQuery.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashShuffleFileWriteExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnTajoResourceManager.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashJoinExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
          • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcServer.java
          • tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashFullOuterJoinExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ProjectionExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #65 (See https://builds.apache.org/job/Tajo-master-build/65/ ) TAJO-598 : Refactoring Tajo RPC. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e2a7dffdb652c77dd35a917134cef15b3d54d274 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcServer.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnContainerProxy.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/UnaryPhysicalExec.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BinaryPhysicalExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/metrics/reporter/TajoMetricsScheduledReporter.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-rpc/src/test/java/org/apache/tajo/rpc/TestBlockingRpc.java tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcClient.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/TPCH.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestGlobalPlanner.java tajo-rpc/src/main/java/org/apache/tajo/rpc/RpcConnectionPool.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java tajo-rpc/src/main/java/org/apache/tajo/rpc/RpcChannelFactory.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java tajo-client/src/main/java/org/apache/tajo/client/TajoClient.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/StoreTableExec.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BNLJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeJoinExec.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MergeFullOuterJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/metrics/reporter/MetricsFileScheduledReporter.java tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyServerBase.java tajo-rpc/src/test/java/org/apache/tajo/rpc/TestAsyncRpc.java tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcClient.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/YarnTaskRunnerLauncherImpl.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/benchmark/SimpleQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerClientService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashShuffleFileWriteExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashAggregateExec.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoContainerProxy.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RightOuterMergeJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnTajoResourceManager.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcServer.java tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashFullOuterJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/ProjectionExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/MemSortExec.java

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              jhkim Jinho Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development