Uploaded image for project: 'HCatalog'
  1. HCatalog
  2. HCATALOG-183

Memory leak in HCat 0.1/0.2

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.2
    • Fix Version/s: None
    • Component/s: metastore
    • Labels:

      Description

      When one leaves the HCatalog server running for long (in a secure setup), with requests continuously coming in, one sees that the memory footprint of the metastore-server increases continuously, until it culminates in an OutOfMemoryError:

      <backtrace>
      2011-12-01 18:11:00,620 ERROR api.ThriftHiveMetastore$Processor (ThriftHiveMetastore.java:process(5949)) - Internal error processing get_partition_names
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2882)
      at java.lang.StringValue.from(StringValue.java:24)
      at java.lang.String.<init>(String.java:178)
      at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:286)
      at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:262)
      at com.mysql.jdbc.ResultSet.getStringInternal(ResultSet.java:5671)
      at com.mysql.jdbc.ResultSet.getString(ResultSet.java:5544)
      at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
      at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
      at org.datanucleus.store.rdbms.mapping.CharRDBMSMapping.getObject(CharRDBMSMapping.java:460)
      at org.datanucleus.store.mapped.mapping.SingleFieldMapping.getObject(SingleFieldMapping.java:216)
      at org.datanucleus.store.rdbms.query.ResultClassROF.processScalarExpression(ResultClassROF.java:583)
      at org.datanucleus.store.rdbms.query.ResultClassROF.getObject(ResultClassROF.java:361)
      at org.datanucleus.store.rdbms.query.legacy.LegacyForwardQueryResult.nextResultSetElement(LegacyForwardQueryResult.java:137)
      at org.datanucleus.store.rdbms.query.legacy.LegacyForwardQueryResult$QueryResultIterator.next(LegacyForwardQueryResult.java:305)
      at org.apache.hadoop.hive.metastore.ObjectStore.listPartitionNames(ObjectStore.java:1200)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$26.run(HiveMetaStore.java:1555)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$26.run(HiveMetaStore.java:1552)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:309)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_names(HiveMetaStore.java:1552)
      ...
      </backtrace>

      The OOM is preceded by other failures, including a "GSS initiate failure" (in spite of a client-side kinit), and an "Error occurred during processing of request".

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mithun Mithun Radhakrishnan
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated: