[HIVE-28450] Follow the array size of JVM in Hive transferable objects

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.3, 4.0.0
• Fix Version/s: 4.1.0
• Component/s: Metastore
• Assignee: Unassigned
• Reporter: Sercan Tekin
• Created: 19/Aug/24 10:13
• Updated: 11/Nov/24 14:49

Description

We are experiencing an issue with a partitioned table in Hive. When querying the table via the Hive CLI, the data retrieval works as expected without any errors. However, when attempting to query the same table through Spark, we encounter the following error in the HMS logs:

2024-01-30 23:03:59,052 main DEBUG org.apache.logging.log4j.core.util.SystemClock does not support precise timestamps.
Exception in thread "pool-7-thread-4" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:473)
	at org.apache.thrift.transport.TSaslServerTransport.write(TSaslServerTransport.java:42)
	at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:227)
	at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.write(FieldSchema.java:517)
	at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.write(FieldSchema.java:456)
	at org.apache.hadoop.hive.metastore.api.FieldSchema.write(FieldSchema.java:394)
	at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1423)
	at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1250)
	at org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1116)
	at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1033)
	at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:890)
	at org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:786)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.write(ThriftHiveMetastore.java)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.write(ThriftHiveMetastore.java)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.write(ThriftHiveMetastore.java)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:58)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:603)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:600)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:600)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Exception in thread "pool-7-thread-6" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Exception in thread "pool-7-thread-9" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

This error appears to be related to the JVM’s conservative approach to array size allocation, which limits the maximum size of arrays to prevent OutOfMemoryError exceptions. For reference, you can see a similar implementation in the JVM source code here: https://github.com/openjdk/jdk/blob/0e0dfca21f64ecfcb3e5ed7cdc2a173834faa509/src/java.base/share/classes/java/io/InputStream.java#L307-L313

Spark side implemented similar limit on their side, it would be good to implement the same thing on Hive side - https://github.com/apache/spark/blob/e5a5921968c84601ce005a7785bdd08c41a2d862/common/utils/src/main/scala/org/apache/spark/unsafe/array/ByteArrayUtils.java

Workaround:
As a temporary workaround, I have been able to mitigate the issue by setting the hive.metastore.batch.retrieve.table.partition.max configuration to a lower value.

Issue Links

• relates to: HIVE-6179 OOM occurs when query spans to a large number of partitions
• relates to: THRIFT-3914 TSaslServerTransport throws OOM due to BetaArrayOutputStream limitation