Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21799

NullPointerException in DynamicPartitionPruningOptimization, when join key is on aggregation column

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: Query Planning
    • Labels:
      None

      Description

      Following table/query results in NPE:

      create table tez_no_dynpart_hashjoin_on_agg(id int, outcome string, eventid int) stored as orc;
      
      explain select a.id, b.outcome from (select id, max(eventid) as event_id_max from tez_no_dynpart_hashjoin_on_agg group by id) a 
      LEFT OUTER JOIN tez_no_dynpart_hashjoin_on_agg b 
      on a.event_id_max = b.eventid;
      

      Stack trace:

      java.lang.NullPointerException
              at org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.generateSemiJoinOperatorPlan(DynamicPartitionPruningOptimization.java:608)
              at org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.process(DynamicPartitionPruningOptimization.java:239)
              at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
              at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
              at org.apache.hadoop.hive.ql.parse.TezCompiler.runDynamicPartitionPruning(TezCompiler.java:584)
              at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:165)
              at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:159)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12562)
              at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:370)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:671)
              at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1905)
              at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1852)
              at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1847)
              at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
              at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:219)
              at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242)
              at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:189)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:408)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:340)
              at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:676)
              at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:647)
              at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:182)
              at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
              at org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:59)
      

        Attachments

        1. HIVE-21799.1.patch
          8 kB
          Jason Dere
        2. HIVE-21799.2.patch
          9 kB
          Jason Dere
        3. HIVE-21799.3.patch
          9 kB
          Jason Dere
        4. HIVE-21799.4.patch
          13 kB
          Jason Dere
        5. HIVE-21799.5.patch
          14 kB
          Jason Dere
        6. HIVE-21799.6.patch
          14 kB
          Jason Dere
        7. HIVE-21799.7.patch
          14 kB
          Jason Dere

          Activity

            People

            • Assignee:
              jdere Jason Dere
              Reporter:
              jdere Jason Dere
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: