Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9893

HiveServer2 java.lang.OutOfMemoryError: Java heap space

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.14.0
    • Fix Version/s: None
    • Component/s: HiveServer2
    • Labels:
    • Environment:

      Vmware Pseudo cluster
      HDP 2.2 Ambari vanilla install
      Centos 6.5

    • Tags:
      Hive Hadoop Tez Thrift HiveServer2

      Description

      Everything runs but dies after a few days with the Java heap space memory error - and with no activity on cluster. It failed most resently after 5 days.

      I tried to fix it after noticing the SLF4J library duplication, wondering if conflicting libraries would be causing the error, but still fails.
      Nagios reports HiveServer2 "Flapping"
      HiveServer2.log output:
      22488-2015-03-07 22:19:08,359 INFO [HiveServer2-Handler-Pool: Thread-13139]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
      22489-2015-03-07 22:19:19,056 ERROR [HiveServer2-Handler-Pool: Thread-13139]: thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing OpenSession
      22490:java.lang.OutOfMemoryError: Java heap space
      22491-2015-03-07 22:19:22,515 INFO [Thread-6]: server.HiveServer2 (HiveServer2.java:stop(299)) - Shutting down HiveServer2
      22492-2015-03-07 22:19:22,516 INFO [Thread-6]: thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped
      22493-2015-03-07 22:19:22,516 INFO [Thread-6]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
      22494-2015-03-07 22:19:27,078 INFO [Thread-6]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
      22495-2015-03-07 22:19:27,078 INFO [Thread-6]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
      22496:2015-03-07 22:19:36,096 WARN [Thread-0]: util.ShutdownHookManager (ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed, java.lang.OutOfMemoryError: Java heap space
      22497:java.lang.OutOfMemoryError: Java heap space

        Activity

        Hide
        rupert160 Rupert Bailey added a comment -

        Could this be a duplicate of:
        https://issues.apache.org/jira/browse/HIVE-7353
        ?

        Show
        rupert160 Rupert Bailey added a comment - Could this be a duplicate of: https://issues.apache.org/jira/browse/HIVE-7353 ?
        Hide
        rupert160 Rupert Bailey added a comment -

        Actually I don't think it is - I'm using 0.14 and HIVE-7353 is for fixing 0.13

        Show
        rupert160 Rupert Bailey added a comment - Actually I don't think it is - I'm using 0.14 and HIVE-7353 is for fixing 0.13
        Hide
        nemon Nemon Lou added a comment -

        Do you see any failed queries? If so,give HIVE-9839 a try.

        Show
        nemon Nemon Lou added a comment - Do you see any failed queries? If so,give HIVE-9839 a try.
        Hide
        fsclavo Fernando Sclavo added a comment -

        Exactly same here, hiveserver runs fine for about a week and then fails with following error in hiverserver2.log:

        2015-05-09 11:18:58,968 INFO [HiveServer2-Handler-Pool: Thread-19209]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
        2015-05-09 11:20:31,085 INFO [HiveServer2-Handler-Pool: Thread-11901]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
        2015-05-09 11:22:20,078 ERROR [HiveServer2-Handler-Pool: Thread-20680]: server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
        java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:190)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 4 more
        2015-05-09 11:24:52,488 ERROR [HiveServer2-Handler-Pool: Thread-19209]: thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing OpenSession
        java.lang.OutOfMemoryError: Java heap space
        2015-05-09 11:25:02,446 ERROR [HiveServer2-Handler-Pool: Thread-30918]: server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message.

        Show
        fsclavo Fernando Sclavo added a comment - Exactly same here, hiveserver runs fine for about a week and then fails with following error in hiverserver2.log: 2015-05-09 11:18:58,968 INFO [HiveServer2-Handler-Pool: Thread-19209] : thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6 2015-05-09 11:20:31,085 INFO [HiveServer2-Handler-Pool: Thread-11901] : thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6 2015-05-09 11:22:20,078 ERROR [HiveServer2-Handler-Pool: Thread-20680] : server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:190) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 4 more 2015-05-09 11:24:52,488 ERROR [HiveServer2-Handler-Pool: Thread-19209] : thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing OpenSession java.lang.OutOfMemoryError: Java heap space 2015-05-09 11:25:02,446 ERROR [HiveServer2-Handler-Pool: Thread-30918] : server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
        Hide
        jacksonhu jackonhu added a comment -

        hi,I'm using CDH-5.3.1,some times the same error occurs.
        2015-07-12 08:35:47,979 ERROR hive.log: Got exception: org.apache.thrift.TApplicationException get_databases failed: out of sequence response
        org.apache.thrift.TApplicationException: get_databases failed: out of sequence response
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837)
        at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        2015-07-12 08:36:33,934 INFO org.apache.hive.service.cli.thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
        2015-07-12 08:37:59,327 WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error getting schemas:
        java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:84)
        at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
        at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
        at java.security.AccessController.doPrivileged(Native Method)

        Show
        jacksonhu jackonhu added a comment - hi,I'm using CDH-5.3.1,some times the same error occurs. 2015-07-12 08:35:47,979 ERROR hive.log: Got exception: org.apache.thrift.TApplicationException get_databases failed: out of sequence response org.apache.thrift.TApplicationException: get_databases failed: out of sequence response at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:600) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:587) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:837) at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2015-07-12 08:36:33,934 INFO org.apache.hive.service.cli.thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6 2015-07-12 08:37:59,327 WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error getting schemas: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:84) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) at java.security.AccessController.doPrivileged(Native Method)
        Hide
        jacksonhu jackonhu added a comment -

        I'm wondering when will hiveserver need so much java heap space ? in error log,I see there are just some getSchemas method call such as getSchemas(CLIService.java:277),recv_get_databases(ThriftHiveMetastore.java:600), how could it take so much space?

        Show
        jacksonhu jackonhu added a comment - I'm wondering when will hiveserver need so much java heap space ? in error log,I see there are just some getSchemas method call such as getSchemas(CLIService.java:277),recv_get_databases(ThriftHiveMetastore.java:600), how could it take so much space?
        Hide
        caseybrown89@gmail.com Casey Brown added a comment -

        Getting the same problem here on EMR,

        Hive 0.13.1
        Subversion http://aws.amazon.com/elasticmapreduce -r 4fad79b21cc5cc0d2a42a9a6e94c3c0bf81cb404
        Compiled by EMR Team on Mon Oct 27 16:56:19 GMT 2014
        From source with checksum 83544a51266f73ac453641ea3932c726

        I'm running a ton of alter table add/remove partitions using pyhs2. These errors crop up after a few hundred of them run. I can restart hiveserver2 and everything is OK for a little while. It shows up to the client as

        conn = pyhs2.connect( authMechanism='PLAIN', user='hadoop', password='', database='default')
        File "/usr/lib/python2.7/site-packages/pyhs2/_init_.py", line 7, in connect
        return Connection(*args, **kwargs)
        File "/usr/lib/python2.7/site-packages/pyhs2/connections.py", line 47, in _init_
        res = self.client.OpenSession(TOpenSessionReq(username=user, password=password, configuration=configuration))
        File "/usr/lib/python2.7/site-packages/pyhs2/TCLIService/TCLIService.py", line 154, in OpenSession
        return self.recv_OpenSession()
        File "/usr/lib/python2.7/site-packages/pyhs2/TCLIService/TCLIService.py", line 170, in recv_OpenSession
        raise x
        thrift.Thrift.TApplicationException: Internal error processing OpenSession

        In the Hive server log, I see this:

        2015-11-11 16:32:37,556 INFO [pool-5-thread-7]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(179)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V1
        2015-11-11 16:32:46,022 ERROR [pool-5-thread-7]: thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing OpenSession
        java.lang.OutOfMemoryError: Java heap space

        If there's any specific logging you'd like me to look for just let me know.

        Show
        caseybrown89@gmail.com Casey Brown added a comment - Getting the same problem here on EMR, Hive 0.13.1 Subversion http://aws.amazon.com/elasticmapreduce -r 4fad79b21cc5cc0d2a42a9a6e94c3c0bf81cb404 Compiled by EMR Team on Mon Oct 27 16:56:19 GMT 2014 From source with checksum 83544a51266f73ac453641ea3932c726 I'm running a ton of alter table add/remove partitions using pyhs2. These errors crop up after a few hundred of them run. I can restart hiveserver2 and everything is OK for a little while. It shows up to the client as conn = pyhs2.connect( authMechanism='PLAIN', user='hadoop', password='', database='default') File "/usr/lib/python2.7/site-packages/pyhs2/_ init _.py", line 7, in connect return Connection(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyhs2/connections.py", line 47, in _ init _ res = self.client.OpenSession(TOpenSessionReq(username=user, password=password, configuration=configuration)) File "/usr/lib/python2.7/site-packages/pyhs2/TCLIService/TCLIService.py", line 154, in OpenSession return self.recv_OpenSession() File "/usr/lib/python2.7/site-packages/pyhs2/TCLIService/TCLIService.py", line 170, in recv_OpenSession raise x thrift.Thrift.TApplicationException: Internal error processing OpenSession In the Hive server log, I see this: 2015-11-11 16:32:37,556 INFO [pool-5-thread-7] : thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(179)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V1 2015-11-11 16:32:46,022 ERROR [pool-5-thread-7] : thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing OpenSession java.lang.OutOfMemoryError: Java heap space If there's any specific logging you'd like me to look for just let me know.

          People

          • Assignee:
            Unassigned
            Reporter:
            rupert160 Rupert Bailey
          • Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development