[HIVE-28207] NullPointerException is thrown when checking column uniqueness

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.1.0, 4.0.1
• Component/s: CBO
• Labels: hive-4.0.1-merged hive-4.0.1-must pull-request-available
• Assignee: Shohei Okumiya
• Reporter: Shohei Okumiya
• Created: 21/Apr/24 11:58
• Updated: 24/Sep/24 02:15
• Resolved: 24/Apr/24 09:46

Description

In some cases, we skip checking null. For example, the last statement in the following set of queries fails with NPE.

CREATE TABLE `store_sales` (`ss_item_sk` bigint);
CREATE TABLE `household_demographics` (`hd_demo_sk` bigint);
CREATE TABLE `item` (`i_item_sk` bigint);
ALTER TABLE `store_sales` ADD CONSTRAINT `pk_ss` PRIMARY KEY (`ss_item_sk`) DISABLE NOVALIDATE RELY;
ALTER TABLE `item` ADD CONSTRAINT `pk_i` PRIMARY KEY (`i_item_sk`) DISABLE NOVALIDATE RELY;
ALTER TABLE `store_sales` ADD CONSTRAINT `ss_i` FOREIGN KEY (`ss_item_sk`) REFERENCES `item`(`i_item_sk`) DISABLE NOVALIDATE RELY;

EXPLAIN
SELECT i_item_sk
FROM store_sales, household_demographics, item
WHERE ss_item_sk = i_item_sk

The NPE happens with HiveJoinConstraintsRule in the above case.

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: NullPointerException null
     at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:214) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.operation.Operation.run(Operation.java:286) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:557) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:542) ~[hive-service-4.0.0.jar:4.0.0]
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
     at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
     at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-4.0.0.jar:4.0.0]
     at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_275]
     at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_275]
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.3.6.jar:?]
     at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-4.0.0.jar:4.0.0]
     at com.sun.proxy.$Proxy42.executeStatementAsync(Unknown Source) ~[?:?]
     at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:316) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:652) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-4.0.0.jar:4.0.0]
     at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[hive-exec-4.0.0.jar:4.0.0]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275]
     at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
Caused by: java.lang.NullPointerException
     at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule.onMatch(HiveJoinConstraintsRule.java:277) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2446) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2405) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1943) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1686) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1567) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1319) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:570) ~[hive-exec-4.0.0.jar:4.0.0]
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13149) ~[hive-exec-4.0.0.jar:4.0.0] 

Issue Links

• links to: GitHub Pull Request #5207