[HIVE-22662] java.lang.NegativeArraySizeException when trying to get HashPartition

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Hive

Environment

Using hdp 3.1.4.0-315

• Assignee: Unassigned
• Reporter: Diego Patricio Moncayo
• Created: 18/Dec/19 15:12
• Updated: 07/Jan/20 14:04

Description

Running a query between a big table and a small one, this error appeared:

Caused by: java.lang.RuntimeException: Map operator initialization failed
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:361)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
	... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async Initialization failed. abortRequested=false
	at org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:465)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:399)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:572)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:524)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:342)
	... 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NegativeArraySizeException
	at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:103)
	at org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NegativeArraySizeException
	at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:271)
	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344)
	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413)
	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215)
	at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96)
	... 5 more
Caused by: java.lang.NegativeArraySizeException
	at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.<init>(HybridHashTableContainer.java:362)
	at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.<init>(HybridHashTableContainer.java:277)
	at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:231)
	... 9 more

 When looking into the stacktrace, this line appeared.

hashPartitions = new HashPartition[numPartitions];

Apparently the int variable numPartition overflows

When this parameter is set hive.auto.convert.join=false the issue disappears 

Similar to this Issue https://issues.apache.org/jira/browse/HIVE-12547 but not the same