[HIVE-16160] OutOfMemoryError: GC overhead limit exceeded on Hiveserver2

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: HiveServer2
• Assignee: Sushanth Sowmyan
• Reporter: Kavan Suresh
• Created: 09/Mar/17 22:56
• Updated: 21/Jul/17 18:45
• Resolved: 16/Mar/17 18:42

Description

Hs2 process killed by OOM:

 ERROR [HiveServer2-Handler-Pool: Thread-1361771]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(203)) - java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.String.toLowerCase(String.java:2647)
        at org.datanucleus.Configuration.getInternalNameForProperty(Configuration.java:188)
        at org.datanucleus.ExecutionContextImpl.getProperty(ExecutionContextImpl.java:1012)
        at org.datanucleus.state.AbstractStateManager.updateLevel2CacheForFields(AbstractStateManager.java:979)
        at org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1097)
        at org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4544)
        at org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4199)
        at org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:770)
        at org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:385)
        at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:275)
        at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:107)
        at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:570)
        at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1033)
        at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
        at com.sun.proxy.$Proxy7.getTable(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1915)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1887)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
        at com.sun.proxy.$Proxy12.get_table(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1271)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:178)

Issue Links

• relates to: HIVE-11499 Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs