[HIVE-13239] "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: HiveServer2

Environment

HDP2.3.4
JDK1.8
CentOS 6

• Assignee: Unassigned
• Reporter: Wataru Yukawa
• Created: 09/Mar/16 02:02
• Updated: 25/Apr/16 02:17
• Resolved: 25/Apr/16 02:17

Description

"ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000

HiveServer2 memory leak occurs.

hive query

 FROM hoge_tmp
 INSERT INTO TABLE hoge PARTITION (...)
   SELECT ...   WHERE ...

stacktrace

org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread
        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391)
        at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577)
        at org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310)
        at org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559)
        at org.apache.tez.client.TezClient.start(TezClient.java:395)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1703)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1460)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1096)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
        ... 15 more