Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-522

OutOfMemoryError: unable to create new native thread

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Another exception occurs on the client side when I run a query

      Progress: 0%, response time: 190.197 sec
      com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: unable to create new native thread
      	at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:193)
      	at org.apache.tajo.cli.TajoCli.getQueryResult(TajoCli.java:353)
      	at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:319)
      	at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228)
      	at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735)
      Caused by: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
      	at org.apache.tajo.rpc.BlockingRpcClient.<init>(BlockingRpcClient.java:71)
      	at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54)
      	at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63)
      	at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:188)
      	... 4 more
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:640)
      	at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
      	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
      	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
      	... 8 more
      java.lang.OutOfMemoryError: unable to create new native thread
      2014-01-19 23:12:19,975 WARN  client.TajoClient (TajoClient.java:closeQuery(110)) - Fail to close a QueryMaster connection (qid=q_1390100273039_0012, msg=java.lang.OutOfMemoryError: unable to create new native thread)
      java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
      	at org.apache.tajo.rpc.BlockingRpcClient.<init>(BlockingRpcClient.java:71)
      	at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54)
      	at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63)
      	at org.apache.tajo.client.TajoClient.closeQuery(TajoClient.java:106)
      	at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:323)
      	at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228)
      	at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735)
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:640)
      	at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
      	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
      	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
      	... 7 more
      

      I am not sure why client side need to create quite a lot of threads.

        Attachments

        1. tajo-site.xml
          4 kB
          Min Zhou
        2. TAJO-522.patch
          6 kB
          Hyunsik Choi

          Issue Links

            Activity

              People

              • Assignee:
                hjkim Hyoungjun Kim
                Reporter:
                coderplay Min Zhou
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: