[HIVE-14764] Enabling "hive.metastore.metrics.enabled" throws OOM in HiveMetastore

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Metastore
• Labels: JMX Metrics Monitoring
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 15/Sep/16 09:02
• Updated: 13/Feb/17 16:16
• Resolved: 16/Sep/16 07:33

Description

After running some queries with metrics enabled, metastore starts throwing the following messages.

Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:433)
        at com.mysql.jdbc.PreparedStatement.getInstance(PreparedStatement.java:877)
        at com.mysql.jdbc.ConnectionImpl.clientPrepareStatement(ConnectionImpl.java:1489)
        at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4343)
        at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4242)
        at com.jolbox.bonecp.ConnectionHandle.prepareStatement(ConnectionHandle.java:1024)
        at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:350)
        at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:295)
        at org.datanucleus.store.rdbms.scostore.JoinListStore.listIterator(JoinListStore.java:761)
        ... 36 more
Nested Throwables StackTrace:
java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:433)
        at com.mysql.jdbc.PreparedStatement.getInstance(PreparedStatement.java:877)
        at com.mysql.jdbc.ConnectionImpl.clientPrepareStatement(ConnectionImpl.java:1489)
        at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4343)
        at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4242)
        at com.jolbox.bonecp.ConnectionHandle.prepareStatement(ConnectionHandle.java:1024)
        at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:350)
        at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:295)
        at org.datanucleus.store.rdbms.scostore.JoinListStore.listIterator(JoinListStore.java:761)
        at org.datanucleus.store.rdbms.scostore.AbstractListStore.listIterator(AbstractListStore.java:93)
        at org.datanucleus.store.rdbms.scostore.AbstractListStore.iterator(AbstractListStore.java:83)
        at org.datanucleus.store.types.wrappers.backed.List.loadFromStore(List.java:264)
        at org.datanucleus.store.types.wrappers.backed.List.iterator(List.java:492)
        at org.apache.hadoop.hive.metastore.ObjectStore.convertToFieldSchemas(ObjectStore.java:1199)
        at org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1266)
        at org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1281)
        at org.apache.hadoop.hive.metastore.ObjectStore.convertToTable(ObjectStore.java:1138)
        at org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:2651)
        at org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:6141)

HiveMetastore uses start/end functions for starting/ending the scope in MetricsFactory. In some places in HiveMetastore the function names are not matching causing gradual memory leak in metastore with metrics enabled.

Issue Links

• breaks: HIVE-15679 Some metastore metrics always contains column=null (write_column_statistics, write_partition_column_statistics)