Hive
  1. Hive
  2. HIVE-4118

ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS fails when using fully qualified table name

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11.0, 0.12.0, 0.13.0
    • Fix Version/s: 0.14.0
    • Component/s: Statistics
    • Labels:
      None

      Description

      Computing column stats fails when using fully qualified table name. Issuing a "USE db" and using only the table name succeeds.

      hive -e "ANALYZE TABLE somedb.some_table COMPUTE STATISTICS FOR COLUMNS int_col"
      
      org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Table somedb.some_table for which stats is gathered doesn't exist.)
      	at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2201)
      	at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:325)
      	at org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:336)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
      	at $Proxy9.updateTableColumnStatistics(Unknown Source)
      	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.update_table_column_statistics(HiveMetaStore.java:3171)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
      	at $Proxy10.update_table_column_statistics(Unknown Source)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.updateTableColumnStatistics(HiveMetaStoreClient.java:973)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
      	at $Proxy11.updateTableColumnStatistics(Unknown Source)
      	at org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2198)
      	... 18 more
      
      
      1. HIVE-4118.1.patch
        0.8 kB
        Bing Li
      2. HIVE-4118.2.patch
        0.8 kB
        Bing Li

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        Duplicate of HIVE-4119

        Show
        Ashutosh Chauhan added a comment - Duplicate of HIVE-4119
        Hide
        Ashutosh Chauhan added a comment -

        Sorry, its not a dupe. Reopening.

        Show
        Ashutosh Chauhan added a comment - Sorry, its not a dupe. Reopening.
        Hide
        Bing Li added a comment -

        Hi, Shreepadma Venugopalan
        I ran into this error in Hive-0.12.0, will you plan to fix this?

        Show
        Bing Li added a comment - Hi, Shreepadma Venugopalan I ran into this error in Hive-0.12.0, will you plan to fix this?
        Hide
        Bing Li added a comment -

        Generated HIVE-4118.1.patch against the trunk

        Show
        Bing Li added a comment - Generated HIVE-4118 .1.patch against the trunk
        Hide
        Szehon Ho added a comment -

        Posting manually

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12645512/HIVE-4118.1.patch

        ERROR: -1 due to 18 failed/errored test(s), 5450 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20
        org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
        org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/233/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/233/console

        Messages:

        
        

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited

        Show
        Szehon Ho added a comment - Posting manually Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12645512/HIVE-4118.1.patch ERROR: -1 due to 18 failed/errored test(s), 5450 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.ql.parse.TestParse.testParse_input20 org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4 org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5 org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/233/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/233/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12645512/HIVE-4118.1.patch

        ERROR: -1 due to 10 failed/errored test(s), 5451 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_multi_distinct
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
        org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
        org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/246/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/246/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logsPreCommit-HIVE-Build-246/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 10 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12645512

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12645512/HIVE-4118.1.patch ERROR: -1 due to 10 failed/errored test(s), 5451 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_multi_distinct org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/246/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/246/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logsPreCommit-HIVE-Build-246/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12645512
        Hide
        Sushanth Sowmyan added a comment -

        Removing 0.13.1 tag on fix version, since the tag implies that this patch has been committed to 0.13.1, which it has not.

        Show
        Sushanth Sowmyan added a comment - Removing 0.13.1 tag on fix version, since the tag implies that this patch has been committed to 0.13.1, which it has not.
        Hide
        Bing Li added a comment -

        Re-created the patch against the latest trunk.

        Show
        Bing Li added a comment - Re-created the patch against the latest trunk.
        Hide
        Bing Li added a comment -

        Re-assign this JIRA to myself.

        Show
        Bing Li added a comment - Re-assign this JIRA to myself.
        Hide
        Bing Li added a comment -

        HIVE-4118.2.patch is generated based on the latest trunk

        Show
        Bing Li added a comment - HIVE-4118 .2.patch is generated based on the latest trunk
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12652144/HIVE-4118.2.patch

        ERROR: -1 due to 5 failed/errored test(s), 5669 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/577/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/577/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-577/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 5 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12652144

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652144/HIVE-4118.2.patch ERROR: -1 due to 5 failed/errored test(s), 5669 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/577/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/577/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-577/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12652144
        Hide
        Ashutosh Chauhan added a comment -

        This I believe got fixed as a by-product of HIVE-7168. Feel free to reopen, if not so.

        Show
        Ashutosh Chauhan added a comment - This I believe got fixed as a by-product of HIVE-7168 . Feel free to reopen, if not so.
        Hide
        Thejas M Nair added a comment -

        This has been fixed in 0.14 release. Please open new jira if you see any issues.

        Show
        Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

          People

          • Assignee:
            Bing Li
            Reporter:
            Lenni Kuff
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development