Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-138

Too many RPC connections in TajoWorker

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Worker
    • Labels:
      None

      Description

      We can see that there are many same connection/disconnection messages from TaskWorker log as the following. This is caused by the fact that each TaskRunner connects to QueryMaster. We need to share those connections among TaskRunners in the same TajoWorker.

      2013-08-29 10:34:51,880 INFO  worker.TajoWorker (TajoWorker.java:run(400)) - Worker Resource Heartbeat Thread stopped.
      2013-08-29 10:34:51,881 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.115:9005
      2013-08-29 10:34:51,884 INFO  rpc.NettyServerBase (NettyServerBase.java:shutdown(120)) - Rpc (TajoWorkerProtocol) listened on 0:0:0:0:0:0:0:0:8092) shutdown
      2013-08-29 10:34:51,884 INFO  worker.TajoWorkerManagerService (TajoWorkerManagerService.java:stop(108)) - TajoWorkerManagerService stopped
      2013-08-29 10:34:51,884 INFO  querymaster.QueryMaster (QueryMaster.java:run(304)) - QueryMaster heartbeat thread stopped
      2013-08-29 10:34:51,884 INFO  service.AbstractService (AbstractService.java:stop(113)) - Service:org.apache.tajo.master.TajoAsyncDispatcher is stopped.
      2013-08-29 10:34:51,885 INFO  master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:querymaster_1377740005035
      2013-08-29 10:34:51,885 INFO  service.AbstractService (AbstractService.java:stop(113)) - Service:org.apache.tajo.master.querymaster.QueryMaster is stopped.
      2013-08-29 10:34:51,885 INFO  querymaster.QueryMaster (QueryMaster.java:stop(146)) - QueryMaster stop
      2013-08-29 10:34:51,885 INFO  service.AbstractService (AbstractService.java:stop(113)) - Service:org.apache.tajo.worker.TajoWorkerManagerService is stopped.
      2013-08-29 10:34:51,885 INFO  worker.TajoWorkerClientService (TajoWorkerClientService.java:stop(103)) - ====> TajoWorkerClientService stopping
      2013-08-29 10:34:51,885 INFO  rpc.NettyServerBase (NettyServerBase.java:shutdown(120)) - Rpc (QueryMasterClientProtocol) listened on 0:0:0:0:0:0:0:0:8091) shutdown
      2013-08-29 10:34:51,885 INFO  worker.TajoWorkerClientService (TajoWorkerClientService.java:stop(107)) - ====> TajoWorkerClientService stopped
      2013-08-29 10:34:51,885 INFO  service.AbstractService (AbstractService.java:stop(113)) - Service:org.apache.tajo.worker.TajoWorkerClientService is stopped.
      2013-08-29 10:34:51,889 INFO  service.AbstractService (AbstractService.java:stop(113)) - Service:httpshuffle is stopped.
      2013-08-29 10:34:51,890 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,890 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,891 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,891 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,892 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,892 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,892 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,892 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,894 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,894 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,895 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,895 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,896 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,896 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,897 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,897 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,898 INFO  rpc.NettyClientBase (NettyClientBase.java:close(87)) - Proxy is disconnected from 211.239.116.119:8092
      2013-08-29 10:34:51,898 INFO  worker.TaskRunner (TaskRunner.java:stop(236)) - Stop TaskRunner: eb_1377740003810_0001_000002
      2013-08-29 10:34:51,898 INFO  service.AbstractService (AbstractService.java:stop(113)) - Service:org.apache.tajo.worker.TaskRunnerManager is stopped.
      2013-08-29 10:34:51,898 INFO  service.AbstractService (AbstractService.java:stop(113)) - Service:org.apache.tajo.worker.TajoWorker is stopped.
      2013-08-29 10:34:51,898 INFO  worker.TajoWorker (TajoWorker.java:stop(187)) - TajoWorker main thread exiting
      
      

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        This issue was resolved by TAJO-274.

        Show
        hyunsik Hyunsik Choi added a comment - This issue was resolved by TAJO-274 .
        Hide
        hjkim Hyoungjun Kim added a comment -

        In order to resolve this issue Connection Pool required. I will implement connection pool for tajo rpc.

        Show
        hjkim Hyoungjun Kim added a comment - In order to resolve this issue Connection Pool required. I will implement connection pool for tajo rpc.

          People

          • Assignee:
            Unassigned
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development