[TAJO-522] OutOfMemoryError: unable to create new native thread

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.8.0
• Assignee: Hyoungjun Kim
• Reporter: Min Zhou
• Created: 19/Jan/14 23:36
• Updated: 23/Jan/14 20:23
• Resolved: 21/Jan/14 01:49

Description

Another exception occurs on the client side when I run a query

Progress: 0%, response time: 190.197 sec
com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:193)
	at org.apache.tajo.cli.TajoCli.getQueryResult(TajoCli.java:353)
	at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:319)
	at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228)
	at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735)
Caused by: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
	at org.apache.tajo.rpc.BlockingRpcClient.<init>(BlockingRpcClient.java:71)
	at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54)
	at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63)
	at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:188)
	... 4 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:640)
	at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
	... 8 more
java.lang.OutOfMemoryError: unable to create new native thread
2014-01-19 23:12:19,975 WARN  client.TajoClient (TajoClient.java:closeQuery(110)) - Fail to close a QueryMaster connection (qid=q_1390100273039_0012, msg=java.lang.OutOfMemoryError: unable to create new native thread)
java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
	at org.apache.tajo.rpc.BlockingRpcClient.<init>(BlockingRpcClient.java:71)
	at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54)
	at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63)
	at org.apache.tajo.client.TajoClient.closeQuery(TajoClient.java:106)
	at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:323)
	at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228)
	at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:640)
	at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
	... 7 more

I am not sure why client side need to create quite a lot of threads.

Issue Links

• is related to: TAJO-537 After TAJO-522, still OutOfMemoryError: unable to create new native thread
• is related to: TAJO-544 Thread pool abusing
• relates to: TAJO-521 org.apache.tajo.rpc.NettyClientBase throws IOE
• relates to: TAJO-523 ExecuttionBlock hangs
• relates to: TAJO-525 Concurrent queries hang