Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-522

OutOfMemoryError: unable to create new native thread

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Another exception occurs on the client side when I run a query

      Progress: 0%, response time: 190.197 sec
      com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: unable to create new native thread
      	at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:193)
      	at org.apache.tajo.cli.TajoCli.getQueryResult(TajoCli.java:353)
      	at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:319)
      	at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228)
      	at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735)
      Caused by: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
      	at org.apache.tajo.rpc.BlockingRpcClient.<init>(BlockingRpcClient.java:71)
      	at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54)
      	at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63)
      	at org.apache.tajo.client.TajoClient.getQueryStatus(TajoClient.java:188)
      	... 4 more
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:640)
      	at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
      	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
      	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
      	... 8 more
      java.lang.OutOfMemoryError: unable to create new native thread
      2014-01-19 23:12:19,975 WARN  client.TajoClient (TajoClient.java:closeQuery(110)) - Fail to close a QueryMaster connection (qid=q_1390100273039_0012, msg=java.lang.OutOfMemoryError: unable to create new native thread)
      java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:76)
      	at org.apache.tajo.rpc.BlockingRpcClient.<init>(BlockingRpcClient.java:71)
      	at org.apache.tajo.rpc.RpcConnectionPool.makeConnection(RpcConnectionPool.java:54)
      	at org.apache.tajo.rpc.RpcConnectionPool.getConnection(RpcConnectionPool.java:63)
      	at org.apache.tajo.client.TajoClient.closeQuery(TajoClient.java:106)
      	at org.apache.tajo.cli.TajoCli.executeStatements(TajoCli.java:323)
      	at org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:228)
      	at org.apache.tajo.cli.TajoCli.main(TajoCli.java:735)
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:640)
      	at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
      	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
      	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
      	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
      	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
      	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
      	at org.apache.tajo.rpc.NettyClientBase.init(NettyClientBase.java:54)
      	... 7 more
      

      I am not sure why client side need to create quite a lot of threads.

      1. TAJO-522.patch
        6 kB
        Hyunsik Choi
      2. tajo-site.xml
        4 kB
        Min Zhou

        Issue Links

          Activity

          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Min,

          Actually, I couldn't reproduce it in my environment. Could you share some of your environment information?
          'ulimit -a' and 'tajo.worker.resource.memory-mb' in tajo-site.xml would be helpful to find the cause of problems.

          Show
          hyunsik Hyunsik Choi added a comment - Hi Min, Actually, I couldn't reproduce it in my environment. Could you share some of your environment information? 'ulimit -a' and 'tajo.worker.resource.memory-mb' in tajo-site.xml would be helpful to find the cause of problems.
          Hide
          coderplay Min Zhou added a comment -

          This line is reported by tsql client, not the worker.

          Here is the result of ulimit -a

          core file size          (blocks, -c) 0
          data seg size           (kbytes, -d) unlimited
          scheduling priority             (-e) 0
          file size               (blocks, -f) unlimited
          pending signals                 (-i) 256476
          max locked memory       (kbytes, -l) 64
          max memory size         (kbytes, -m) unlimited
          open files                      (-n) 65536
          pipe size            (512 bytes, -p) 8
          POSIX message queues     (bytes, -q) 819200
          real-time priority              (-r) 0
          stack size              (kbytes, -s) 10240
          cpu time               (seconds, -t) unlimited
          max user processes              (-u) 10000
          virtual memory          (kbytes, -v) unlimited
          file locks                      (-x) unlimited
          

          I will submit my tajo-site.xml soon

          Show
          coderplay Min Zhou added a comment - This line is reported by tsql client, not the worker. Here is the result of ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256476 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65536 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 10000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited I will submit my tajo-site.xml soon
          Hide
          hyunsik Hyunsik Choi added a comment -

          Could you show me conf/tajo-env.sh?

          I wonder that you set a proper memory size to worker heap memory.
          http://tajo.incubator.apache.org/tajo-0.8.0-doc.html#TajoMasterHeap

          Show
          hyunsik Hyunsik Choi added a comment - Could you show me conf/tajo-env.sh? I wonder that you set a proper memory size to worker heap memory. http://tajo.incubator.apache.org/tajo-0.8.0-doc.html#TajoMasterHeap
          Hide
          hyunsik Hyunsik Choi added a comment -

          This patch is moved from TAJO-525.

          Show
          hyunsik Hyunsik Choi added a comment - This patch is moved from TAJO-525 .
          Hide
          hyunsik Hyunsik Choi added a comment -

          According to Min's reports in TAJO-525, we can expect that this problem is essentially caused by one bug that creates too many threads in TajoContainerProxy::assignExecutionBlock.

          Show
          hyunsik Hyunsik Choi added a comment - According to Min's reports in TAJO-525 , we can expect that this problem is essentially caused by one bug that creates too many threads in TajoContainerProxy::assignExecutionBlock.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12623913/TAJO-522.patch
          against master revision d04f9a5.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 181 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core/tajo-core-backend tajo-rpc.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/65//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/65//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/65//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-rpc.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/65//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/65//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12623913/TAJO-522.patch against master revision d04f9a5. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 181 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core/tajo-core-backend tajo-rpc. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/65//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/65//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/65//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-rpc.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/65//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/65//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1

          Thank you for the quick fix.

          Show
          hyunsik Hyunsik Choi added a comment - +1 Thank you for the quick fix.
          Hide
          hyunsik Hyunsik Choi added a comment -

          I've just commit the latest patch submitted in RB. Also, I appreciate Min for the detailed report. It was very helpful to find the problem and fix.

          Show
          hyunsik Hyunsik Choi added a comment - I've just commit the latest patch submitted in RB. Also, I appreciate Min for the detailed report. It was very helpful to find the problem and fix.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #21 (See https://builds.apache.org/job/Tajo-master-build/21/)
          TAJO-522: OutOfMemoryError: unable to create new native thread. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=948915151b14774c606bba7a78751600d2368261)

          • tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • CHANGES.txt
          • tajo-rpc/pom.xml
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #21 (See https://builds.apache.org/job/Tajo-master-build/21/ ) TAJO-522 : OutOfMemoryError: unable to create new native thread. (hyoungjunkim via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=948915151b14774c606bba7a78751600d2368261 ) tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java CHANGES.txt tajo-rpc/pom.xml tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
          Hide
          coderplay Min Zhou added a comment -

          Hi all, I ran a big query applied this patch, still OOM. Please submit this patch and I will the next step.

          Show
          coderplay Min Zhou added a comment - Hi all, I ran a big query applied this patch, still OOM. Please submit this patch and I will the next step.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Is your OOM the same phenomenon reported here?

          Show
          hyunsik Hyunsik Choi added a comment - Is your OOM the same phenomenon reported here?
          Hide
          coderplay Min Zhou added a comment -

          Yes, still create so many threads. As I mentioned on the reviewboard, this patch can significantly alleviate the native memory pressure of worker. However, if the quantity of different connections is large, this patch won't help.

          Show
          coderplay Min Zhou added a comment - Yes, still create so many threads. As I mentioned on the reviewboard, this patch can significantly alleviate the native memory pressure of worker. However, if the quantity of different connections is large, this patch won't help.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #22 (See https://builds.apache.org/job/Tajo-master-build/22/)
          TAJO-537: After TAJO-522, still OutOfMemoryError: unable to create new native thread. (Min Zhou via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e57cf426d8d1d1b924a32a10f81a8912f2b0c45f)

          • tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcClient.java
          • CHANGES.txt
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcClient.java
          • tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #22 (See https://builds.apache.org/job/Tajo-master-build/22/ ) TAJO-537 : After TAJO-522 , still OutOfMemoryError: unable to create new native thread. (Min Zhou via hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=e57cf426d8d1d1b924a32a10f81a8912f2b0c45f ) tajo-rpc/src/main/java/org/apache/tajo/rpc/AsyncRpcClient.java CHANGES.txt tajo-rpc/src/main/java/org/apache/tajo/rpc/BlockingRpcClient.java tajo-rpc/src/main/java/org/apache/tajo/rpc/NettyClientBase.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java

            People

            • Assignee:
              hjkim Hyoungjun Kim
              Reporter:
              coderplay Min Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development