Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18577

SemanticAnalyzer.validate has some pointless metastore calls

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Transactions
    • None

    Attachments

      1. HIVE-18577.patch
        4 kB
        Sergey Shelukhin

      Activity

        ashutoshc can you take a look? these metastore calls look completely useless to me cause the code just above seems to access partition objects from the WriteEntity just fine.

        sershe Sergey Shelukhin added a comment - ashutoshc can you take a look? these metastore calls look completely useless to me cause the code just above seems to access partition objects from the WriteEntity just fine.

        +1

        ashutoshc Ashutosh Chauhan added a comment - +1

        HIVE-11145 removed Protect mode because of same reason that this is expensive metadata call, but stopped short of removing corresponding logic which it should had.

        +1

        ashutoshc Ashutosh Chauhan added a comment - HIVE-11145 removed Protect mode because of same reason that this is expensive metadata call, but stopped short of removing corresponding logic which it should had. +1
        hiveqa Hive QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
              Prechecks
        0 findbugs 0m 0s Findbugs executables are not available.
        +1 @author 0m 0s The patch does not contain any @author tags.
              master Compile Tests
        +1 mvninstall 7m 9s master passed
        +1 compile 1m 4s master passed
        +1 checkstyle 0m 46s master passed
        +1 javadoc 0m 53s master passed
              Patch Compile Tests
        +1 mvninstall 1m 17s the patch passed
        +1 compile 1m 4s the patch passed
        +1 javac 1m 4s the patch passed
        -1 checkstyle 0m 43s ql: The patch generated 1 new + 780 unchanged - 1 fixed = 781 total (was 781)
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 javadoc 0m 51s the patch passed
              Other Tests
        +1 asflicense 0m 12s The patch does not generate ASF License warnings.
        14m 15s



        Subsystem Report/Notes
        Optional Tests asflicense javac javadoc findbugs checkstyle compile
        uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux
        Build tool maven
        Personality /data/hiveptest/working/yetus/dev-support/hive-personality.sh
        git revision master / be8a451
        Default Java 1.8.0_111
        checkstyle http://104.198.109.242/logs//PreCommit-HIVE-Build-8920/yetus/diff-checkstyle-ql.txt
        modules C: ql U: ql
        Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-8920/yetus.txt
        Powered by Apache Yetus http://yetus.apache.org

        This message was automatically generated.

        hiveqa Hive QA added a comment - -1 overall Vote Subsystem Runtime Comment       Prechecks 0 findbugs 0m 0s Findbugs executables are not available. +1 @author 0m 0s The patch does not contain any @author tags.       master Compile Tests +1 mvninstall 7m 9s master passed +1 compile 1m 4s master passed +1 checkstyle 0m 46s master passed +1 javadoc 0m 53s master passed       Patch Compile Tests +1 mvninstall 1m 17s the patch passed +1 compile 1m 4s the patch passed +1 javac 1m 4s the patch passed -1 checkstyle 0m 43s ql: The patch generated 1 new + 780 unchanged - 1 fixed = 781 total (was 781) +1 whitespace 0m 0s The patch has no whitespace issues. +1 javadoc 0m 51s the patch passed       Other Tests +1 asflicense 0m 12s The patch does not generate ASF License warnings. 14m 15s Subsystem Report/Notes Optional Tests asflicense javac javadoc findbugs checkstyle compile uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux Build tool maven Personality /data/hiveptest/working/yetus/dev-support/hive-personality.sh git revision master / be8a451 Default Java 1.8.0_111 checkstyle http://104.198.109.242/logs//PreCommit-HIVE-Build-8920/yetus/diff-checkstyle-ql.txt modules C: ql U: ql Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-8920/yetus.txt Powered by Apache Yetus http://yetus.apache.org This message was automatically generated.
        hiveqa Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12908252/HIVE-18577.patch

        ERROR: -1 due to no test(s) being added or modified.

        ERROR: -1 due to 21 failed/errored test(s), 12779 tests executed
        Failed tests:

        TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=150)
        	[intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,table_nonprintable.q,tez_union_dynamic_partition.q,tez_union_dynamic_partition_2.q,temp_table_external.q,global_limit.q,llap_udf.q,schemeAuthority.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,parallel_colstats.q]
        org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175)
        org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152)
        org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172)
        org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167)
        org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171)
        org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161)
        org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164)
        org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161)
        org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes] (batchId=163)
        org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122)
        org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221)
        org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282)
        org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
        org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
        org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
        org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
        org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8920/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8920/console
        Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8920/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.YetusPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 21 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12908252 - PreCommit-HIVE-Build

        hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12908252/HIVE-18577.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 21 failed/errored test(s), 12779 tests executed Failed tests: TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=150) [intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,table_nonprintable.q,tez_union_dynamic_partition.q,tez_union_dynamic_partition_2.q,temp_table_external.q,global_limit.q,llap_udf.q,schemeAuthority.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,parallel_colstats.q] org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes] (batchId=163) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8920/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8920/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8920/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed This message is automatically generated. ATTACHMENT ID: 12908252 - PreCommit-HIVE-Build

        Failures that are not timeouts are definitely unrelated. I'll try to rerun the timed out test locally, logs seem to only have some move errors on localhost so probably a test env issue.

        sershe Sergey Shelukhin added a comment - Failures that are not timeouts are definitely unrelated. I'll try to rerun the timed out test locally, logs seem to only have some move errors on localhost so probably a test env issue.

        Committed to master. Thanks for the review!

        sershe Sergey Shelukhin added a comment - Committed to master. Thanks for the review!

        This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

        ashutoshc Ashutosh Chauhan added a comment - This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

        People

          sershe Sergey Shelukhin
          sershe Sergey Shelukhin
          Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved: