Hive
  1. Hive
  2. HIVE-4561

Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, 0.13.0
    • Fix Version/s: 0.14.0
    • Component/s: Statistics
    • Labels:
      None

      Description

      if all column values larger than 0.0 DOUBLE_LOW_VALUE always will be 0.0
      or if all column values less than 0.0, DOUBLE_HIGH_VALUE will always be

      hive (default)> create table src_test (price double);
      hive (default)> load data local inpath './test.txt' into table src_test;
      hive (default)> select * from src_test;
      OK
      1.0
      2.0
      3.0
      Time taken: 0.313 seconds, Fetched: 3 row(s)
      hive (default)> analyze table src_test compute statistics for columns price;

      mysql> select * from TAB_COL_STATS \G;
      CS_ID: 16
      DB_NAME: default
      TABLE_NAME: src_test
      COLUMN_NAME: price
      COLUMN_TYPE: double
      TBL_ID: 2586
      LONG_LOW_VALUE: 0
      LONG_HIGH_VALUE: 0
      DOUBLE_LOW_VALUE: 0.0000 # Wrong Result ! Expected is 1.0000
      DOUBLE_HIGH_VALUE: 3.0000
      BIG_DECIMAL_LOW_VALUE: NULL
      BIG_DECIMAL_HIGH_VALUE: NULL
      NUM_NULLS: 0
      NUM_DISTINCTS: 1
      AVG_COL_LEN: 0.0000
      MAX_COL_LEN: 0
      NUM_TRUES: 0
      NUM_FALSES: 0
      LAST_ANALYZED: 1368596151
      2 rows in set (0.00 sec)

      1. HIVE-4561.9.patch.txt
        133 kB
        Navis
      2. HIVE-4561.8.patch.txt
        157 kB
        Navis
      3. HIVE-4561.7.patch.txt
        132 kB
        Navis
      4. HIVE-4561.6.patch.txt
        131 kB
        Navis
      5. HIVE-4561.5.patch.txt
        129 kB
        Navis
      6. HIVE-4561.4.patch.txt
        79 kB
        Navis
      7. HIVE-4561.3.patch
        3 kB
        Zhuoluo (Clark) Yang
      8. HIVE-4561.2.patch
        2 kB
        Zhuoluo (Clark) Yang
      9. HIVE-4561.1.patch
        2 kB
        Zhuoluo (Clark) Yang

        Issue Links

          Activity

          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Navis!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis!
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12647283/HIVE-4561.9.patch.txt

          ERROR: -1 due to 8 failed/errored test(s), 5467 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.common.metrics.TestMetrics.testScopeConcurrency
          org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/340/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/340/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-340/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12647283

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12647283/HIVE-4561.9.patch.txt ERROR: -1 due to 8 failed/errored test(s), 5467 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.common.metrics.TestMetrics.testScopeConcurrency org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/340/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/340/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-340/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12647283
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12647045/HIVE-4561.8.patch.txt

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/323/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/323/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-323/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
          + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-323/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'common/src/test/org/apache/hadoop/hive/common/type/TestHiveDecimal.java'
          Reverted 'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestHiveSchemaConverter.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java'
          ++ awk '{print $2}'
          ++ egrep -v '^X|^Performing status on external'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target data/files/dec_comp.txt itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1598190.
          
          At revision 1598190.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12647045

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12647045/HIVE-4561.8.patch.txt Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/323/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/323/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-323/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-323/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'common/src/test/org/apache/hadoop/hive/common/type/TestHiveDecimal.java' Reverted 'ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestHiveSchemaConverter.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java' ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target data/files/dec_comp.txt itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1598190. At revision 1598190. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12647045
          Hide
          Ashutosh Chauhan added a comment -

          Can you also update RB with latest patch ?

          Show
          Ashutosh Chauhan added a comment - Can you also update RB with latest patch ?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12646836/HIVE-4561.7.patch.txt

          ERROR: -1 due to 16 failed/errored test(s), 5466 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4_multi_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_decimal
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_count
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
          org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/313/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/313/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-313/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 16 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12646836

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12646836/HIVE-4561.7.patch.txt ERROR: -1 due to 16 failed/errored test(s), 5466 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup4_multi_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_decimal org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_count org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/313/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/313/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-313/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed This message is automatically generated. ATTACHMENT ID: 12646836
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Thanks, Navis, It seems more situations needs to be consider.

          Show
          Zhuoluo (Clark) Yang added a comment - Thanks, Navis , It seems more situations needs to be consider.
          Hide
          Navis added a comment -

          Ok, I'll take a look.

          Show
          Navis added a comment - Ok, I'll take a look.
          Hide
          Ashutosh Chauhan added a comment -

          Both testCliDriver_columnstats_tbllvl.q & testCliDriver_compute_stats_empty_table.q failed for me on latest patch with following exception. which is same as Hive QA run reported on http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_columnstats_tbllvl/

          java.lang.Exception: java.lang.RuntimeException: Hive Runtime Error while closing operators
                  at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
                  at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
          Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
                  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:260)
                  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
                  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
                  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
                  at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
                  at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1141)
                  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
                  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
                  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
                  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
                  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
                  ... 9 more
          Caused by: java.lang.NullPointerException
                  at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
                  at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:598)
                  at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFLongStatsEvaluator.iterate(GenericUDAFComputeStats.java:468)
                  at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:183)
                  at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1132)
          
          Show
          Ashutosh Chauhan added a comment - Both testCliDriver_columnstats_tbllvl.q & testCliDriver_compute_stats_empty_table.q failed for me on latest patch with following exception. which is same as Hive QA run reported on http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_columnstats_tbllvl/ java.lang.Exception: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:260) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1141) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591) at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227) ... 9 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36) at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:598) at org.apache.hadoop.hive.ql.udf. generic .GenericUDAFComputeStats$GenericUDAFLongStatsEvaluator.iterate(GenericUDAFComputeStats.java:468) at org.apache.hadoop.hive.ql.udf. generic .GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:183) at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1132)
          Hide
          Navis added a comment -

          Fixed result of metadata_only_queries_with_filters.q, but cannot reproduce other failures. Rerunning test.

          Show
          Navis added a comment - Fixed result of metadata_only_queries_with_filters.q, but cannot reproduce other failures. Rerunning test.
          Hide
          Ashutosh Chauhan added a comment -

          Failures need to be looked at

          Show
          Ashutosh Chauhan added a comment - Failures need to be looked at
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12646662/HIVE-4561.5.patch.txt

          ERROR: -1 due to 11 failed/errored test(s), 5462 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_tbllvl
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_compute_stats_empty_table
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries_with_filters
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-291/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 11 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12646662

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12646662/HIVE-4561.5.patch.txt ERROR: -1 due to 11 failed/errored test(s), 5462 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_tbllvl org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_compute_stats_empty_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries_with_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_decimal1 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/291/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-291/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed This message is automatically generated. ATTACHMENT ID: 12646662
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Ashutosh Chauhan added a comment -

          metadata_only_queries.q update seems to be correct now. But, annotate_stats_groupby.q & annotate_stats_table.q seems like failures with NPE that needs to be looked at.

          Show
          Ashutosh Chauhan added a comment - metadata_only_queries.q update seems to be correct now. But, annotate_stats_groupby.q & annotate_stats_table.q seems like failures with NPE that needs to be looked at.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12645973/HIVE-4561.4.patch.txt

          ERROR: -1 due to 13 failed/errored test(s), 5451 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.conf.TestHiveConf.testConfProperties
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/259/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/259/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-259/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 13 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12645973

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12645973/HIVE-4561.4.patch.txt ERROR: -1 due to 13 failed/errored test(s), 5451 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.conf.TestHiveConf.testConfProperties org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/259/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/259/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-259/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed This message is automatically generated. ATTACHMENT ID: 12645973
          Hide
          Navis added a comment -

          Changed to allow null for min/max value of long/double/decimal stats. Let's see what will happen.

          Show
          Navis added a comment - Changed to allow null for min/max value of long/double/decimal stats. Let's see what will happen.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12586100/HIVE-4561.3.patch

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/951/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/951/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-951/source-prep.txt
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          ++ awk '{print $2}'
          ++ egrep -v '^X|^Performing status on external'
          ++ svn status --no-ignore
          + rm -rf
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1527155.
          
          At revision 1527155.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0 to p2
          + exit 1
          '
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12586100/HIVE-4561.3.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/951/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/951/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-951/source-prep.txt + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1527155. At revision 1527155. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0 to p2 + exit 1 ' This message is automatically generated.
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Shreepadma Venugopalan It seems null value will involve a lot of modification, such as ColumnStatsTask or ObjectStore or thrift files.
          Currently, statistics are looked up by code, I think it make sense to keep Long.Min/Long.Max.
          And we can apply HIVE-4561.3.patch instead of HIVE-4561.4.patch.

          Show
          Zhuoluo (Clark) Yang added a comment - Shreepadma Venugopalan It seems null value will involve a lot of modification, such as ColumnStatsTask or ObjectStore or thrift files. Currently, statistics are looked up by code, I think it make sense to keep Long.Min/Long.Max. And we can apply HIVE-4561 .3.patch instead of HIVE-4561 .4.patch.
          Hide
          Shreepadma Venugopalan added a comment -

          Zhuoluo (Clark) Yang: My suggestion is to use Long.Min/Long.Max value instead of a null value. The code that looks at column stats can use the min/max in conjunction with other stats such as number of rows etc. to infer that the values are initialization values for min/max and not true values that represent the bounds on the column.

          Show
          Shreepadma Venugopalan added a comment - Zhuoluo (Clark) Yang : My suggestion is to use Long.Min/Long.Max value instead of a null value. The code that looks at column stats can use the min/max in conjunction with other stats such as number of rows etc. to infer that the values are initialization values for min/max and not true values that represent the bounds on the column.
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Shreepadma Venugopalan I think I am wrong. Originally, I want to "return" like this:

          @@ -189,6 +187,11 @@
                   statsObj.setStatsData(statsData);
                 }
               } else {
          +      // Any null object, such as min/max value of an empty table,
          +      // need not be unpacked.
          +      if (o == null) {
          +        return;
          +      }
                 // invoke the right unpack method depending on data type of the column
                 if (statsObj.getStatsData().isSetBooleanStats()) {
                   unpackBooleanStats(oi, o, fieldName, statsObj);
          

          However, I've found that LongColumnStatsData.highValue is required by thrift. And also modifications of ObjectStore is required and checks LongColumnStatsData.isSetHighValue(). Any suggestions? Thanks!

          Show
          Zhuoluo (Clark) Yang added a comment - Shreepadma Venugopalan I think I am wrong. Originally, I want to "return" like this: @@ -189,6 +187,11 @@ statsObj.setStatsData(statsData); } } else { + // Any null object, such as min/max value of an empty table, + // need not be unpacked. + if (o == null ) { + return ; + } // invoke the right unpack method depending on data type of the column if (statsObj.getStatsData().isSetBooleanStats()) { unpackBooleanStats(oi, o, fieldName, statsObj); However, I've found that LongColumnStatsData.highValue is required by thrift. And also modifications of ObjectStore is required and checks LongColumnStatsData.isSetHighValue(). Any suggestions? Thanks!
          Hide
          Shreepadma Venugopalan added a comment -

          Zhuoluo (Clark) Yang: I'm not sure I understand the fix here. Can you please elaborate on what it means to leaving it empty in the ColumnStatsTask? Thanks!

          Show
          Shreepadma Venugopalan added a comment - Zhuoluo (Clark) Yang : I'm not sure I understand the fix here. Can you please elaborate on what it means to leaving it empty in the ColumnStatsTask? Thanks!
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Ashutosh Chauhan I think it happens when we try to persist a null max/min,I think the simplest way is to leave it empty in the ColumnStatsTask. I will try to make a new patch and make a full test.

          Show
          Zhuoluo (Clark) Yang added a comment - Ashutosh Chauhan I think it happens when we try to persist a null max/min,I think the simplest way is to leave it empty in the ColumnStatsTask. I will try to make a new patch and make a full test.
          Hide
          Ashutosh Chauhan added a comment -

          Now, columnstats_tbllvl.q failed with following exception:

             [junit] java.lang.NullPointerException
              [junit]     at org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyLongObjectInspector.get(LazyLongObjectInspector.java:38)
              [junit]     at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.unpackLongStats(ColumnStatsTask.java:126)
              [junit]     at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.unpackPrimitiveObject(ColumnStatsTask.java:196)
              [junit]     at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.unpackStructObject(ColumnStatsTask.java:224)
              [junit]     at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.constructColumnStatsFromPackedRow(ColumnStatsTask.java:263)
              [junit]     at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:327)
              [junit]     at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:343)
              [junit]     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145)
              [junit]     at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
              [junit]     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1355)
              [junit]     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1139)
              [junit]     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:945)
              [junit]     at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
              [junit]     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
              [junit]     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
              [junit]     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
              [junit]     at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790)
              [junit]     at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:6279)
              [junit]     at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_tbllvl(TestCliDriver.java:1971)
          
          Show
          Ashutosh Chauhan added a comment - Now, columnstats_tbllvl.q failed with following exception: [junit] java.lang.NullPointerException [junit] at org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyLongObjectInspector.get(LazyLongObjectInspector.java:38) [junit] at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.unpackLongStats(ColumnStatsTask.java:126) [junit] at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.unpackPrimitiveObject(ColumnStatsTask.java:196) [junit] at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.unpackStructObject(ColumnStatsTask.java:224) [junit] at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.constructColumnStatsFromPackedRow(ColumnStatsTask.java:263) [junit] at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:327) [junit] at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:343) [junit] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:145) [junit] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) [junit] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1355) [junit] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1139) [junit] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:945) [junit] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) [junit] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) [junit] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) [junit] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) [junit] at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:6279) [junit] at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_tbllvl(TestCliDriver.java:1971)
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Update patch, make HIGH/LOW values of empty tables return null.

          Show
          Zhuoluo (Clark) Yang added a comment - Update patch, make HIGH/LOW values of empty tables return null.
          Hide
          Ashutosh Chauhan added a comment -

          If there are no rows in table, low and high values should really be null (or NaN) and not 0.

          Show
          Ashutosh Chauhan added a comment - If there are no rows in table, low and high values should really be null (or NaN) and not 0.
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Ashutosh Chauhan The values sounds quite strange, I will try to make a new patch.

          Show
          Zhuoluo (Clark) Yang added a comment - Ashutosh Chauhan The values sounds quite strange, I will try to make a new patch.
          Hide
          Ashutosh Chauhan added a comment -

          Is that correct? For empty table "min":9223372036854775807,"max":-9223372036854775808 doesn't sound right.

          Show
          Ashutosh Chauhan added a comment - Is that correct? For empty table "min":9223372036854775807,"max":-9223372036854775808 doesn't sound right.
          Hide
          Zhuoluo (Clark) Yang added a comment -

          fix compute_stats_empty_table.q test results.

          Show
          Zhuoluo (Clark) Yang added a comment - fix compute_stats_empty_table.q test results.
          Hide
          Ashutosh Chauhan added a comment -

          Test compute_stats_empty_table.q failed.

          Show
          Ashutosh Chauhan added a comment - Test compute_stats_empty_table.q failed.
          Hide
          Shreepadma Venugopalan added a comment -

          LGTM! +1 (non-binding).

          Show
          Shreepadma Venugopalan added a comment - LGTM! +1 (non-binding).
          Hide
          Shreepadma Venugopalan added a comment -

          Ashutosh Chauhan: Sure, I'll take a look at this today.

          Show
          Shreepadma Venugopalan added a comment - Ashutosh Chauhan : Sure, I'll take a look at this today.
          Hide
          Ashutosh Chauhan added a comment -

          Shreepadma Venugopalan Since you wrote this originally, would you like to review this as well ?

          Show
          Ashutosh Chauhan added a comment - Shreepadma Venugopalan Since you wrote this originally, would you like to review this as well ?
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Hi, Ashutosh Chauhan, Thanks for your comments. I've updated a new patch on the reveiwboard.

          Show
          Zhuoluo (Clark) Yang added a comment - Hi, Ashutosh Chauhan , Thanks for your comments. I've updated a new patch on the reveiwboard.
          Hide
          Zhuoluo (Clark) Yang added a comment -

          Update a new patch.
          In case of all the long values are positive, we can get the right min. In case of all the values are negative, we can get the right max.
          UT "compute_stats_long.q" reads values from data/files/int.txt which values are all above zero. Original ut computes the min value "0", however, the correct min value is "4". This patch fixes the bug.

          Show
          Zhuoluo (Clark) Yang added a comment - Update a new patch. In case of all the long values are positive, we can get the right min. In case of all the values are negative, we can get the right max. UT "compute_stats_long.q" reads values from data/files/int.txt which values are all above zero. Original ut computes the min value "0", however, the correct min value is "4". This patch fixes the bug.
          Hide
          Ashutosh Chauhan added a comment -

          Assigned to Zhuoluo (Clark) Yang. Can you also take a look at review comments?

          Show
          Ashutosh Chauhan added a comment - Assigned to Zhuoluo (Clark) Yang . Can you also take a look at review comments?
          Show
          Zhuoluo (Clark) Yang added a comment - https://reviews.apache.org/r/11172/
          Hide
          Zhuoluo (Clark) Yang added a comment -

          A quick fix.

          Show
          Zhuoluo (Clark) Yang added a comment - A quick fix.
          Hide
          Zhuoluo (Clark) Yang added a comment -

          A quick fix, would any body assign the issue to me?

          Show
          Zhuoluo (Clark) Yang added a comment - A quick fix, would any body assign the issue to me?

            People

            • Assignee:
              Navis
              Reporter:
              caofangkun
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development