[HCATALOG-183] Memory leak in HCat 0.1/0.2

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.2
• Component/s: metastore
• Labels: OOM thrift
• Assignee: Unassigned
• Reporter: Mithun Radhakrishnan
• Created: 16/Dec/11 07:13
• Updated: 12/Jan/12 19:41

Description

When one leaves the HCatalog server running for long (in a secure setup), with requests continuously coming in, one sees that the memory footprint of the metastore-server increases continuously, until it culminates in an OutOfMemoryError:

<backtrace>
2011-12-01 18:11:00,620 ERROR api.ThriftHiveMetastore$Processor (ThriftHiveMetastore.java:process(5949)) - Internal error processing get_partition_names
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.StringValue.from(StringValue.java:24)
at java.lang.String.<init>(String.java:178)
at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:286)
at com.mysql.jdbc.SingleByteCharsetConverter.toString(SingleByteCharsetConverter.java:262)
at com.mysql.jdbc.ResultSet.getStringInternal(ResultSet.java:5671)
at com.mysql.jdbc.ResultSet.getString(ResultSet.java:5544)
at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)
at org.datanucleus.store.rdbms.mapping.CharRDBMSMapping.getObject(CharRDBMSMapping.java:460)
at org.datanucleus.store.mapped.mapping.SingleFieldMapping.getObject(SingleFieldMapping.java:216)
at org.datanucleus.store.rdbms.query.ResultClassROF.processScalarExpression(ResultClassROF.java:583)
at org.datanucleus.store.rdbms.query.ResultClassROF.getObject(ResultClassROF.java:361)
at org.datanucleus.store.rdbms.query.legacy.LegacyForwardQueryResult.nextResultSetElement(LegacyForwardQueryResult.java:137)
at org.datanucleus.store.rdbms.query.legacy.LegacyForwardQueryResult$QueryResultIterator.next(LegacyForwardQueryResult.java:305)
at org.apache.hadoop.hive.metastore.ObjectStore.listPartitionNames(ObjectStore.java:1200)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$26.run(HiveMetaStore.java:1555)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$26.run(HiveMetaStore.java:1552)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:309)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_names(HiveMetaStore.java:1552)
...
</backtrace>

The OOM is preceded by other failures, including a "GSS initiate failure" (in spite of a client-side kinit), and an "Error occurred during processing of request".

Issue Links

• is blocked by: HIVE-2715 Upgrade Thrift dependency to 0.9.0
• is blocked by: THRIFT-1468 Memory leak in TSaslServerTransport