[HIVE-24073] Execution exception in sort-merge semijoin

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Operators
• Labels: pull-request-available
• Assignee: mahesh kumar behera
• Reporter: Jesús Camacho Rodríguez
• Created: 25/Aug/20 21:55
• Updated: 27/Feb/24 22:13
• Resolved: 08/Jun/21 02:07

Description

Working on HIVE-24041, we trigger an additional SJ conversion that leads to this exception at execution time:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Attempting to overwrite nextKeyWritables[1]
	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1063)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:685)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:707)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:707)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:707)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:707)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:462)
	... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Attempting to overwrite nextKeyWritables[1]
	at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1037)
	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1060)
	... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Attempting to overwrite nextKeyWritables[1]
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.processKey(CommonMergeJoinOperator.java:564)
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:243)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:887)
	at org.apache.hadoop.hive.ql.exec.TezDummyStoreOperator.process(TezDummyStoreOperator.java:49)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:887)
	at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1003)
	at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1020)
	... 23 more

To reproduce, just set hive.auto.convert.sortmerge.join to true in the last query in auto_sortmerge_join_10.q after HIVE-24041 has been merged.

Issue Links

• relates to: HIVE-24041 Extend semijoin conversion rules
• links to: GitHub Pull Request #1476