[HIVE-18148] NPE in SparkDynamicPartitionPruningResolver

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Spark
• Assignee: Rui Li
• Reporter: Rui Li
• Created: 27/Nov/17 02:38
• Updated: 22/May/18 23:17
• Resolved: 12/Jan/18 03:16

Description

The stack trace is:

2017-11-27T10:32:38,752 ERROR [e6c8aab5-ddd2-461d-b185-a7597c3e7519 main] ql.Driver: FAILED: NullPointerException null
java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver$SparkDynamicPartitionPruningDispatcher.dispatch(SparkDynamicPartitionPruningResolver.java:100)
        at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
        at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
        at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
        at org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver.resolve(SparkDynamicPartitionPruningResolver.java:74)
        at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeTaskPlan(SparkCompiler.java:568)

At this stage, there shouldn't be a DPP sink whose target map work is null. The root cause seems to be a malformed operator tree generated by SplitOpTreeForDPP.

Issue Links

• links to: RB link