Hive
  1. Hive
  2. HIVE-7396

BucketingSortingReduceSinkOptimizer throws NullPointException during ETL

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      When a bucketed insert is performed from a partitioned table into a new table

      2014-07-11 22:11:30,176 ERROR [main]: ql.Driver (SessionState.java:printError(660)) - FAILED: NullPointerException null
      java.lang.NullPointerException
              at org.apache.hadoop.hive.ql.optimizer.BucketingSortingReduceSinkOptimizer$BucketSortReduceSinkProcessor.process(BucketingSortingReduceSinkOptimizer.java:501)
              at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
              at org.apache.hadoop.hive.ql.optimizer.BucketingSortingReduceSinkOptimizer.transform(BucketingSortingReduceSinkOptimizer.java:97)
              at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9503)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:328)
      

        Activity

        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Navis!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis!
        Hide
        Ashutosh Chauhan added a comment -

        +1

        Show
        Ashutosh Chauhan added a comment - +1
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12655367/HIVE-7396.1.patch.txt

        SUCCESS: +1 5719 tests passed

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/760/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/760/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-760/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12655367

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12655367/HIVE-7396.1.patch.txt SUCCESS: +1 5719 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/760/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/760/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-760/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12655367
        Hide
        Navis added a comment -

        My bad

        Show
        Navis added a comment - My bad
        Hide
        Gopal V added a comment -

        This is not just a simple NULL check missing.

        I made modifications to test this out

        --- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
        +++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
        @@ -498,7 +498,7 @@ public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx,
         
                     if (srcTable.isPartitioned()) {
                       PrunedPartitionList prunedParts = pGraphContext.getOpToPartList().get(ts);
        -              List<Partition> partitions = prunedParts.getNotDeniedPartns();
        +              List<Partition> partitions = (prunedParts != null ) ? prunedParts.getNotDeniedPartns() : null;
         
                       // Support for dynamic partitions can be added later
                       // The following is not optimized:
        

        But then it generates a scan on only one partition instead of a full-table scan.

        Show
        Gopal V added a comment - This is not just a simple NULL check missing. I made modifications to test this out --- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java +++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java @@ -498,7 +498,7 @@ public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, if (srcTable.isPartitioned()) { PrunedPartitionList prunedParts = pGraphContext.getOpToPartList().get(ts); - List<Partition> partitions = prunedParts.getNotDeniedPartns(); + List<Partition> partitions = (prunedParts != null ) ? prunedParts.getNotDeniedPartns() : null ; // Support for dynamic partitions can be added later // The following is not optimized: But then it generates a scan on only one partition instead of a full-table scan.

          People

          • Assignee:
            Navis
            Reporter:
            Gopal V
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development