Hive
  1. Hive
  2. HIVE-4119

ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails with NPE if the table is empty

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11.0
    • Component/s: Statistics
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails with NPE if the table is empty

      hive -e "create table empty_table (i int); select compute_stats(i, 16) from empty_table"
      
      
      java.lang.NullPointerException
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:35)
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:535)
      	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFLongStatsEvaluator.iterate(GenericUDAFComputeStats.java:477)
      	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:139)
      	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1099)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1132)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:35)
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:535)
      	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFLongStatsEvaluator.iterate(GenericUDAFComputeStats.java:477)
      	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:139)
      	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1099)
      	... 15 more
      org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1132)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)
      	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:35)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:35)
      	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:535)
      	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFLongStatsEvaluator.iterate(GenericUDAFComputeStats.java:477)
      	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:139)
      	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1099)
      	... 15 more
      
      
      1. HIVE-4119.1.patch
        28 kB
        Shreepadma Venugopalan
      2. HIVE-4119.2.patch
        28 kB
        Shreepadma Venugopalan

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/)
        HIVE-4119. ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails with NPE if the table is empty (Shreepadma Venugopalan via cws) (Revision 1464208)

        Result = FAILURE
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464208
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
        • /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_empty_table.q
        • /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_empty_table.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/ ) HIVE-4119 . ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails with NPE if the table is empty (Shreepadma Venugopalan via cws) (Revision 1464208) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464208 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_empty_table.q /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_empty_table.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #2044 (See https://builds.apache.org/job/Hive-trunk-h0.21/2044/)
        HIVE-4119. ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails with NPE if the table is empty (Shreepadma Venugopalan via cws) (Revision 1464208)

        Result = FAILURE
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464208
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
        • /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_empty_table.q
        • /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_empty_table.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #2044 (See https://builds.apache.org/job/Hive-trunk-h0.21/2044/ ) HIVE-4119 . ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails with NPE if the table is empty (Shreepadma Venugopalan via cws) (Revision 1464208) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464208 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_empty_table.q /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_empty_table.q.out
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Shreepadma!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Shreepadma!
        Hide
        Ashutosh Chauhan added a comment -

        In process of wrapping up 0.11. Looks like this is ready to go in. Will commit if tests pass.

        Show
        Ashutosh Chauhan added a comment - In process of wrapping up 0.11. Looks like this is ready to go in. Will commit if tests pass.
        Hide
        Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        Show
        Carl Steinbach added a comment - +1. Will commit if tests pass.
        Hide
        Shreepadma Venugopalan added a comment -

        Carl Steinbach: Would it be possible to take a look at the new patch? Thanks.

        Show
        Shreepadma Venugopalan added a comment - Carl Steinbach : Would it be possible to take a look at the new patch? Thanks.
        Hide
        Shreepadma Venugopalan added a comment -

        New patch addresses the review comments. Thanks.

        Show
        Shreepadma Venugopalan added a comment - New patch addresses the review comments. Thanks.
        Hide
        Carl Steinbach added a comment -

        The patch looks good with the exception of some minor formatting issues which I noted on reviewboard.

        Show
        Carl Steinbach added a comment - The patch looks good with the exception of some minor formatting issues which I noted on reviewboard.
        Hide
        Shreepadma Venugopalan added a comment -
        Show
        Shreepadma Venugopalan added a comment - Review request: https://reviews.apache.org/r/9929/
        Hide
        Lenni Kuff added a comment -

        This is especially bad because if executing via a Hive Server it will cause the service process to crash.

        Show
        Lenni Kuff added a comment - This is especially bad because if executing via a Hive Server it will cause the service process to crash.

          People

          • Assignee:
            Shreepadma Venugopalan
            Reporter:
            Lenni Kuff
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development