Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18513 Query results caching
  3. HIVE-20242

Query results cache: Improve ability of queries to use pending query results

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • None

    Description

      HIVE-19138 allowed a currently running query to wait on the pending results of an already running query. gopalv, after testing with high concurrency, suggested further improving this by having a way to use the switch to using the results cache even at the end of query compilation.

      Attachments

        1. HIVE-20242.1.patch
          4 kB
          Jason Dere

        Activity

          jdere Jason Dere added a comment -

          Patch performs a 2nd lookup of the cache at the end of query compilation, in case another query has entered pending state by that point.

          jdere Jason Dere added a comment - Patch performs a 2nd lookup of the cache at the end of query compilation, in case another query has entered pending state by that point.
          hiveqa Hive QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
                master Compile Tests
          +1 mvninstall 8m 27s master passed
          +1 compile 1m 8s master passed
          +1 checkstyle 0m 45s master passed
          0 findbugs 4m 11s ql in master has 2296 extant Findbugs warnings.
          +1 javadoc 1m 1s master passed
                Patch Compile Tests
          +1 mvninstall 1m 31s the patch passed
          +1 compile 1m 11s the patch passed
          +1 javac 1m 11s the patch passed
          +1 checkstyle 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 25s the patch passed
          +1 javadoc 1m 0s the patch passed
                Other Tests
          +1 asflicense 0m 13s The patch does not generate ASF License warnings.
          25m 9s



          Subsystem Report/Notes
          Optional Tests asflicense javac javadoc findbugs checkstyle compile
          uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux
          Build tool maven
          Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-12867/dev-support/hive-personality.sh
          git revision master / 9d78fac
          Default Java 1.8.0_111
          findbugs v3.0.0
          modules C: ql U: ql
          Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-12867/yetus.txt
          Powered by Apache Yetus http://yetus.apache.org

          This message was automatically generated.

          hiveqa Hive QA added a comment - +1 overall Vote Subsystem Runtime Comment       Prechecks +1 @author 0m 0s The patch does not contain any @author tags.       master Compile Tests +1 mvninstall 8m 27s master passed +1 compile 1m 8s master passed +1 checkstyle 0m 45s master passed 0 findbugs 4m 11s ql in master has 2296 extant Findbugs warnings. +1 javadoc 1m 1s master passed       Patch Compile Tests +1 mvninstall 1m 31s the patch passed +1 compile 1m 11s the patch passed +1 javac 1m 11s the patch passed +1 checkstyle 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 25s the patch passed +1 javadoc 1m 0s the patch passed       Other Tests +1 asflicense 0m 13s The patch does not generate ASF License warnings. 25m 9s Subsystem Report/Notes Optional Tests asflicense javac javadoc findbugs checkstyle compile uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux Build tool maven Personality /data/hiveptest/working/yetus_PreCommit-HIVE-Build-12867/dev-support/hive-personality.sh git revision master / 9d78fac Default Java 1.8.0_111 findbugs v3.0.0 modules C: ql U: ql Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-12867/yetus.txt Powered by Apache Yetus http://yetus.apache.org This message was automatically generated.
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12933121/HIVE-20242.1.patch

          ERROR: -1 due to no test(s) being added or modified.

          SUCCESS: +1 due to 14812 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12867/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12867/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12867/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.YetusPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12933121 - PreCommit-HIVE-Build

          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12933121/HIVE-20242.1.patch ERROR: -1 due to no test(s) being added or modified. SUCCESS: +1 due to 14812 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12867/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12867/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12867/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12933121 - PreCommit-HIVE-Build

          LGTM - +1

          This cut the number of queries slipping through the compile->pending window by 4x - right now queries which are arriving after 100ms of the 1st query are all getting the pending entry.

          gopalv Gopal Vijayaraghavan added a comment - LGTM - +1 This cut the number of queries slipping through the compile->pending window by 4x - right now queries which are arriving after 100ms of the 1st query are all getting the pending entry.
          jdere Jason Dere added a comment -

          Committed to master

          jdere Jason Dere added a comment - Committed to master

          People

            jdere Jason Dere
            jdere Jason Dere
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: