Hive
  1. Hive
  2. HIVE-5899

NPE during explain extended with char/varchar columns

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Statistics
    • Labels:
      None

      Description

      Running analyze table .. for columns with char/varchar columns and subsequently trying to run explain extended will get a NullPointerException when Hive tries to annotate the operator tree with stats:
      2013-11-26 01:53:06,682 ERROR ql.Driver (SessionState.java:printError(440)) - FAILED: NullPointerException null
      java.lang.NullPointerException
      at org.apache.hadoop.hive.metastore.api.ColumnStatisticsData.getFieldDesc(ColumnStatisticsData.java:367)
      at org.apache.hadoop.hive.metastore.api.ColumnStatisticsData.getStringStats(ColumnStatisticsData.java:444)
      at org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatistics(StatsUtils.java:414)
      at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStatsForColumn(StatsUtils.java:369)
      at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:465)
      at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:109)
      at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:102)
      at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
      at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:54)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
      at org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:76)
      at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:136)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8913)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:292)
      at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:65)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:292)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:341)
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:994)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:905)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
      at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

        Activity

        Jason Dere created issue -
        Hide
        Jason Dere added a comment -

        Couple areas in the stats logic where it fetches the correct stats type based on the type string, which needed to be updated for char/varchar. Uploading patch v1.

        Show
        Jason Dere added a comment - Couple areas in the stats logic where it fetches the correct stats type based on the type string, which needed to be updated for char/varchar. Uploading patch v1.
        Jason Dere made changes -
        Field Original Value New Value
        Attachment HIVE-5899.1.patch [ 12616167 ]
        Jason Dere made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12616167/HIVE-5899.1.patch

        SUCCESS: +1 4740 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/473/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/473/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12616167

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12616167/HIVE-5899.1.patch SUCCESS: +1 4740 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/473/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/473/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12616167
        Hide
        Harish Butani added a comment -

        +1

        Show
        Harish Butani added a comment - +1
        Hide
        Harish Butani added a comment -

        Thanks Jason

        Show
        Harish Butani added a comment - Thanks Jason
        Harish Butani made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Ashutosh Chauhan made changes -
        Fix Version/s 0.13.0 [ 12324986 ]

          People

          • Assignee:
            Jason Dere
            Reporter:
            Jason Dere
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development