[HIVE-27943] NPE in VectorMapJoinCommonOperator.setUpHashTable when running query with join on date

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0-beta-1
• Fix Version/s: 4.0.0
• Component/s: Query Processor
• Labels: pull-request-available
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 07/Dec/23 15:15
• Updated: 29/Mar/24 17:49
• Resolved: 13/Dec/23 10:12

Description

The error can be reproduced on current master using the following steps.

set hive.auto.convert.join=true;

CREATE TABLE person (fname string, birthDate date);
INSERT INTO person VALUES ('Victor', '2023-11-27'), ('Alexandre', '2023-11-28');

SELECT * FROM person p1 INNER JOIN person p2 ON p1.birthDate=p2.birthDate;

The stacktrace is shown below.

2023-12-07T07:06:39,744 ERROR [TezTR-592758_1_2_1_0_0] tez.TezProcessor: Failed initializeAndRunProcessor
java.lang.RuntimeException: Map operator initialization failed
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:351) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:292) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:276) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_261]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_261]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.3.6.jar:?]
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) ~[tez-common-0.10.2.jar:0.10.2]
        at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118) ~[hive-llap-server-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_261]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_261]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_261]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.setUpHashTable(VectorMapJoinCommonOperator.java:673) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.completeInitializationOp(VectorMapJoinCommonOperator.java:634) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:450) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:382) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:549) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:503) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:369) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:332) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        ... 16 more

This is probably caused by HIVE-23852 which added the DATE type as a possible key for the HashTable.

Issue Links

• is caused by: HIVE-23852 Natively support Date type in ReduceSink operator
• links to: GitHub Pull Request #4929