Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12713

Miscellaneous improvements in driver compile and execute logging

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 2.0.0
    • Logging
    • None

    Description

      Miscellaneous compile and execute logging improvements include:
      1. ensuring that only the redacted query to be logged out
      2. removing redundant variable substitution in HS2 SQLOperation
      3. logging out the query and its compilation time without having to enable PerfLogger debug, to help identify badly written queries which take a lot of time to compile and probably cause other good queries to be queued (HIVE-12516)

      Attachments

        1. HIVE-12713.1.patch
          15 kB
          Chaoyu Tang
        2. HIVE-12713.patch
          15 kB
          Chaoyu Tang
        3. HIVE-12713-branch-1.patch
          12 kB
          Chaoyu Tang
        4. HIVE-12713-branch-2.0.patch
          15 kB
          Chaoyu Tang

        Issue Links

          Activity

            ctang Chaoyu Tang added a comment -

            Patch has been uploaded to https://reviews.apache.org/r/41582/ and requested for review. Thanks in advance.

            ctang Chaoyu Tang added a comment - Patch has been uploaded to https://reviews.apache.org/r/41582/ and requested for review. Thanks in advance.
            ctang Chaoyu Tang added a comment -

            Re-uploaded patch with trailing empty spaces removed.

            ctang Chaoyu Tang added a comment - Re-uploaded patch with trailing empty spaces removed.
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12778722/HIVE-12713.patch

            SUCCESS: +1 due to 5 test(s) being added or modified.

            ERROR: -1 due to 20 failed/errored test(s), 9953 tests executed
            Failed tests:

            TestHWISessionManager - did not produce a TEST-*.xml file
            TestMiniTezCliDriver-script_pipe.q-insert_values_non_partitioned.q-subquery_in.q-and-12-more - did not produce a TEST-*.xml file
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata
            org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
            org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
            org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
            org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
            org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
            org.apache.hive.jdbc.TestSSL.testSSLVersion
            org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
            org.apache.hive.spark.client.TestSparkClient.testCounters
            org.apache.hive.spark.client.TestSparkClient.testErrorJob
            org.apache.hive.spark.client.TestSparkClient.testJobSubmission
            org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
            org.apache.hive.spark.client.TestSparkClient.testRemoteClient
            org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
            org.apache.hive.spark.client.TestSparkClient.testSyncRpc
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6420/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6420/console
            Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6420/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 20 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12778722 - PreCommit-HIVE-TRUNK-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12778722/HIVE-12713.patch SUCCESS: +1 due to 5 test(s) being added or modified. ERROR: -1 due to 20 failed/errored test(s), 9953 tests executed Failed tests: TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-script_pipe.q-insert_values_non_partitioned.q-subquery_in.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch org.apache.hive.jdbc.TestSSL.testSSLVersion org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles org.apache.hive.spark.client.TestSparkClient.testCounters org.apache.hive.spark.client.TestSparkClient.testErrorJob org.apache.hive.spark.client.TestSparkClient.testJobSubmission org.apache.hive.spark.client.TestSparkClient.testMetricsCollection org.apache.hive.spark.client.TestSparkClient.testRemoteClient org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob org.apache.hive.spark.client.TestSparkClient.testSyncRpc Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6420/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6420/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6420/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 20 tests failed This message is automatically generated. ATTACHMENT ID: 12778722 - PreCommit-HIVE-TRUNK-Build
            ctang Chaoyu Tang added a comment -

            The failed tests are not related to this patch.

            ctang Chaoyu Tang added a comment - The failed tests are not related to this patch.
            xuefuz Xuefu Zhang added a comment -

            Patch looks good to me. Minor comment/question on RV. Also, it's strange that so many tests failed. Hoping that's a test env issue.

            xuefuz Xuefu Zhang added a comment - Patch looks good to me. Minor comment/question on RV. Also, it's strange that so many tests failed. Hoping that's a test env issue.
            ctang Chaoyu Tang added a comment -

            Thanks xuefuz. I have uploaded a new patch based on the review. The failed tests are all aged and need to be addressed with separate JIRAs.

            ctang Chaoyu Tang added a comment - Thanks xuefuz . I have uploaded a new patch based on the review. The failed tests are all aged and need to be addressed with separate JIRAs.
            xuefuz Xuefu Zhang added a comment -

            +1

            xuefuz Xuefu Zhang added a comment - +1
            hiveqa Hive QA added a comment -

            Here are the results of testing the latest attachment:
            https://issues.apache.org/jira/secure/attachment/12778846/HIVE-12713.1.patch

            SUCCESS: +1 due to 5 test(s) being added or modified.

            ERROR: -1 due to 22 failed/errored test(s), 9955 tests executed
            Failed tests:

            TestHWISessionManager - did not produce a TEST-*.xml file
            TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
            org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata
            org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
            org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
            org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
            org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
            org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
            org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
            org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery
            org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
            org.apache.hive.jdbc.TestSSL.testSSLVersion
            org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
            org.apache.hive.spark.client.TestSparkClient.testCounters
            org.apache.hive.spark.client.TestSparkClient.testErrorJob
            org.apache.hive.spark.client.TestSparkClient.testJobSubmission
            org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
            org.apache.hive.spark.client.TestSparkClient.testRemoteClient
            org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
            org.apache.hive.spark.client.TestSparkClient.testSyncRpc
            

            Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6435/testReport
            Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6435/console
            Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6435/

            Messages:

            Executing org.apache.hive.ptest.execution.TestCheckPhase
            Executing org.apache.hive.ptest.execution.PrepPhase
            Executing org.apache.hive.ptest.execution.ExecutionPhase
            Executing org.apache.hive.ptest.execution.ReportingPhase
            Tests exited with: TestsFailedException: 22 tests failed
            

            This message is automatically generated.

            ATTACHMENT ID: 12778846 - PreCommit-HIVE-TRUNK-Build

            hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12778846/HIVE-12713.1.patch SUCCESS: +1 due to 5 test(s) being added or modified. ERROR: -1 due to 22 failed/errored test(s), 9955 tests executed Failed tests: TestHWISessionManager - did not produce a TEST-*.xml file TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_filemetadata org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_stats_filemetadata org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable org.apache.hive.jdbc.TestSSL.testSSLVersion org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles org.apache.hive.spark.client.TestSparkClient.testCounters org.apache.hive.spark.client.TestSparkClient.testErrorJob org.apache.hive.spark.client.TestSparkClient.testJobSubmission org.apache.hive.spark.client.TestSparkClient.testMetricsCollection org.apache.hive.spark.client.TestSparkClient.testRemoteClient org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob org.apache.hive.spark.client.TestSparkClient.testSyncRpc Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6435/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6435/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6435/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed This message is automatically generated. ATTACHMENT ID: 12778846 - PreCommit-HIVE-TRUNK-Build
            ctang Chaoyu Tang added a comment -

            Two new failed tests (org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery and org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable) could not be reproduced in my local machine and are also not related to this patch.

            ctang Chaoyu Tang added a comment - Two new failed tests (org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery and org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable) could not be reproduced in my local machine and are also not related to this patch.
            ctang Chaoyu Tang added a comment -

            Committed to 1.3.0, 2.0.0 and 2.1.0. Thanks xuefuz for review.

            ctang Chaoyu Tang added a comment - Committed to 1.3.0, 2.0.0 and 2.1.0. Thanks xuefuz for review.

            People

              ctang Chaoyu Tang
              ctang Chaoyu Tang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: