Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1197

Unit test failed: unable to create new native thread

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Unit Test
    • Labels:
      None

      Description

      TravisCI are occasionally failed due to too many thread created

      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.563 sec <<< FAILURE! - in org.apache.tajo.master.ha.TestHAServiceHDFSImpl
      testTwoBackupMasters(org.apache.tajo.master.ha.TestHAServiceHDFSImpl)  Time elapsed: 4.563 sec  <<< ERROR!
      java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:679)
      	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
      	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
      	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
      	at org.apache.tajo.rpc.RpcChannelFactory.createServerChannelFactory(RpcChannelFactory.java:89)
      	at org.apache.tajo.rpc.NettyServerBase.init(NettyServerBase.java:69)
      	at org.apache.tajo.rpc.AsyncRpcServer.<init>(AsyncRpcServer.java:58)
      	at org.apache.tajo.worker.TajoWorkerManagerService.init(TajoWorkerManagerService.java:73)
      	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
      	at org.apache.tajo.worker.TajoWorker.serviceInit(TajoWorker.java:236)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	at org.apache.tajo.worker.TajoWorker.startWorker(TajoWorker.java:146)
      	at org.apache.tajo.TajoTestingCluster.startTajoWorkers(TajoTestingCluster.java:401)
      	at org.apache.tajo.TajoTestingCluster.startMiniTajoCluster(TajoTestingCluster.java:339)
      	at org.apache.tajo.TajoTestingCluster.startMiniCluster(TajoTestingCluster.java:490)
      	at org.apache.tajo.TajoTestingCluster.startMiniCluster(TajoTestingCluster.java:455)
      	at org.apache.tajo.master.ha.TestHAServiceHDFSImpl.testTwoBackupMasters(TestHAServiceHDFSImpl.java:56)
      
      

        Activity

        Hide
        jhkim Jinho Kim added a comment -

        I found some case as following :

        • HA unit test does not close the testing cluster
        • TajoCli test does not close the jline stream
        • Some unit test does not close the TajoClient
        • Default RPC threads is too big in unit test
        Show
        jhkim Jinho Kim added a comment - I found some case as following : HA unit test does not close the testing cluster TajoCli test does not close the jline stream Some unit test does not close the TajoClient Default RPC threads is too big in unit test
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/255

        TAJO-1197: Unit test failed: unable to create new native thread

        I verified as following:
        Before
        Live threads : 600+

        After
        Live threads : 250+

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-1197

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/255.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #255


        commit 67bdc71b17b54da731eaf01b9fac51ad8983353b
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-19T06:46:47Z

        TAJO-1195: Remove unused CachedDNSResolver Class. (DaeMyung Kang via jaehwa)

        Closes #253

        commit b9bed66bcea5eb22c76695e44fbe3c8ca40afed7
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-19T08:12:09Z

        TAJO-1197: Unit test failed: unable to create new native thread


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/255 TAJO-1197 : Unit test failed: unable to create new native thread I verified as following: Before Live threads : 600+ After Live threads : 250+ You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1197 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/255.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #255 commit 67bdc71b17b54da731eaf01b9fac51ad8983353b Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-19T06:46:47Z TAJO-1195 : Remove unused CachedDNSResolver Class. (DaeMyung Kang via jaehwa) Closes #253 commit b9bed66bcea5eb22c76695e44fbe3c8ca40afed7 Author: jhkim <jhkim@apache.org> Date: 2014-11-19T08:12:09Z TAJO-1197 : Unit test failed: unable to create new native thread
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/255#issuecomment-63606149

        Awesome work!
        But, your patch seems to include another commit (TAJO-1195).
        Would you check it, please?

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/255#issuecomment-63606149 Awesome work! But, your patch seems to include another commit ( TAJO-1195 ). Would you check it, please?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/255#issuecomment-63606419

        @jihoonson
        TAJO-1195 was committed by JaeHwa
        It is late mirror sync issue.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/255#issuecomment-63606419 @jihoonson TAJO-1195 was committed by JaeHwa It is late mirror sync issue.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/255#issuecomment-63607587

        @jinossy ok. I missed it.
        +1!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/255#issuecomment-63607587 @jinossy ok. I missed it. +1!
        Hide
        jhkim Jinho Kim added a comment -

        Committed it! Thank you for your quick review!

        Show
        jhkim Jinho Kim added a comment - Committed it! Thank you for your quick review!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #457 (See https://builds.apache.org/job/Tajo-master-build/457/)
        TAJO-1197: Unit test failed: unable to create new native thread. (jinho) (jhkim: rev ef282ebec579c512bbb3860a49d7322e1fb45199)

        • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
        • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
        • tajo-core/src/main/java/org/apache/tajo/scheduler/SimpleFifoScheduler.java
        • tajo-core/src/test/java/org/apache/tajo/master/ha/TestHAServiceHDFSImpl.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
        • tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java
        • tajo-client/src/main/java/org/apache/tajo/cli/tsql/TajoCli.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #457 (See https://builds.apache.org/job/Tajo-master-build/457/ ) TAJO-1197 : Unit test failed: unable to create new native thread. (jinho) (jhkim: rev ef282ebec579c512bbb3860a49d7322e1fb45199) tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/src/main/java/org/apache/tajo/scheduler/SimpleFifoScheduler.java tajo-core/src/test/java/org/apache/tajo/master/ha/TestHAServiceHDFSImpl.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java CHANGES tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java tajo-client/src/main/java/org/apache/tajo/cli/tsql/TajoCli.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #99 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/99/)
        TAJO-1197: Unit test failed: unable to create new native thread. (jinho) (jhkim: rev ef282ebec579c512bbb3860a49d7322e1fb45199)

        • tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core/src/test/java/org/apache/tajo/master/ha/TestHAServiceHDFSImpl.java
        • CHANGES
        • tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
        • tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java
        • tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
        • tajo-client/src/main/java/org/apache/tajo/cli/tsql/TajoCli.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
        • tajo-core/src/main/java/org/apache/tajo/scheduler/SimpleFifoScheduler.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #99 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/99/ ) TAJO-1197 : Unit test failed: unable to create new native thread. (jinho) (jhkim: rev ef282ebec579c512bbb3860a49d7322e1fb45199) tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/test/java/org/apache/tajo/master/ha/TestHAServiceHDFSImpl.java CHANGES tajo-core/src/test/java/org/apache/tajo/cli/tsql/TestTajoCli.java tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-client/src/main/java/org/apache/tajo/cli/tsql/TajoCli.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/src/main/java/org/apache/tajo/scheduler/SimpleFifoScheduler.java
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/255

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/255

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development