Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5240

Column statistics on a partitioned column should fail early with proper error message

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: Statistics
    • Labels:

      Description

      When computing column statistics on a partitioned table, if one of the columns equals the partitioned column then IndexOutOfBoundsException is thrown.

      Following analyze query throws IndexOutOfBoundsException during semantic analysis phase

      hive> analyze table qlog_1m_part partition(year=5) compute statistics for columns year,month,week,type;
      FAILED: IndexOutOfBoundsException Index: 1, Size: 0

      If the partitioned column is specified at last like below then the same exception is thrown at runtime

      hive> analyze table qlog_1m_part partition(year=5) compute statistics for columns month,week,type,year;
      
      Hadoop job information for null: number of mappers: 0; number of reducers: 0
      2013-09-06 18:05:06,587 null map = 0%,  reduce = 100%
      Ended Job = job_local861862820_0001
      Execution completed successfully
      Mapred Local Task Succeeded . Convert the Join into MapJoin
      java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
      	at java.util.LinkedList.entry(LinkedList.java:365)
      	at java.util.LinkedList.get(LinkedList.java:315)
      	at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.constructColumnStatsFromPackedRow(ColumnStatsTask.java:262)
      	at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistPartitionStats(ColumnStatsTask.java:302)
      	at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:345)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1407)
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1187)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1017)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:885)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
      	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
      
      

        Attachments

        1. HIVE-5240.2.txt
          2 kB
          Prasanth Jayachandran
        2. HIVE-5240.3.txt
          3 kB
          Prasanth Jayachandran
        3. HIVE-5240.txt
          2 kB
          Prasanth Jayachandran

          Activity

            People

            • Assignee:
              prasanth_j Prasanth Jayachandran
              Reporter:
              prasanth_j Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: