Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14764

Enabling "hive.metastore.metrics.enabled" throws OOM in HiveMetastore

    XMLWordPrintableJSON

    Details

      Description

      After running some queries with metrics enabled, metastore starts throwing the following messages.

      Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
              at com.mysql.jdbc.Util.handleNewInstance(Util.java:433)
              at com.mysql.jdbc.PreparedStatement.getInstance(PreparedStatement.java:877)
              at com.mysql.jdbc.ConnectionImpl.clientPrepareStatement(ConnectionImpl.java:1489)
              at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4343)
              at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4242)
              at com.jolbox.bonecp.ConnectionHandle.prepareStatement(ConnectionHandle.java:1024)
              at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:350)
              at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:295)
              at org.datanucleus.store.rdbms.scostore.JoinListStore.listIterator(JoinListStore.java:761)
              ... 36 more
      Nested Throwables StackTrace:
      java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1075)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
              at com.mysql.jdbc.Util.handleNewInstance(Util.java:433)
              at com.mysql.jdbc.PreparedStatement.getInstance(PreparedStatement.java:877)
              at com.mysql.jdbc.ConnectionImpl.clientPrepareStatement(ConnectionImpl.java:1489)
              at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4343)
              at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4242)
              at com.jolbox.bonecp.ConnectionHandle.prepareStatement(ConnectionHandle.java:1024)
              at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:350)
              at org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:295)
              at org.datanucleus.store.rdbms.scostore.JoinListStore.listIterator(JoinListStore.java:761)
              at org.datanucleus.store.rdbms.scostore.AbstractListStore.listIterator(AbstractListStore.java:93)
              at org.datanucleus.store.rdbms.scostore.AbstractListStore.iterator(AbstractListStore.java:83)
              at org.datanucleus.store.types.wrappers.backed.List.loadFromStore(List.java:264)
              at org.datanucleus.store.types.wrappers.backed.List.iterator(List.java:492)
              at org.apache.hadoop.hive.metastore.ObjectStore.convertToFieldSchemas(ObjectStore.java:1199)
              at org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1266)
              at org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1281)
              at org.apache.hadoop.hive.metastore.ObjectStore.convertToTable(ObjectStore.java:1138)
              at org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:2651)
              at org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:6141)
      

      HiveMetastore uses start/end functions for starting/ending the scope in MetricsFactory. In some places in HiveMetastore the function names are not matching causing gradual memory leak in metastore with metrics enabled.

        Attachments

        1. hd_1.png
          344 kB
          Rajesh Balamohan
        2. hd_2.png
          386 kB
          Rajesh Balamohan
        3. HIVE-14764.1.patch
          9 kB
          Rajesh Balamohan

          Issue Links

            Activity

              People

              • Assignee:
                rajesh.balamohan Rajesh Balamohan
                Reporter:
                rajesh.balamohan Rajesh Balamohan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: