Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • None
    • None

    Description

      HIVE-17841 already adds a method, it just needs to be called when there are relevant cluster changes that HS2 can detect

      Attachments

        1. HIVE-17905.patch
          10 kB
          Sergey Shelukhin

        Issue Links

        Activity

          sershe Sergey Shelukhin created issue -

          Calling it from an existing background update thread.
          Also added a test.

          sershe Sergey Shelukhin added a comment - Calling it from an existing background update thread. Also added a test.
          sershe Sergey Shelukhin made changes -
          Field Original Value New Value
          Attachment HIVE-17905.patch [ 12899721 ]

          Prasanth Jayachandran can you take a look? small patch

          sershe Sergey Shelukhin added a comment - Prasanth Jayachandran can you take a look? small patch
          sershe Sergey Shelukhin made changes -
          Assignee Sergey Shelukhin [ sershe ]
          Status Open [ 1 ] Patch Available [ 10002 ]

          looks good to me, +1

          Although the core change went in earlier, we can improve the latency of cluster change notification.
          Looks like we are polling for changes every 60s? This seems like a lot where for about a minute we have stale info. Is that right?
          Any reason not to use registry notifications for changes to llap cluster size and then compute executor counts on every ZK notification?

          prasanth_j Prasanth Jayachandran added a comment - looks good to me, +1 Although the core change went in earlier, we can improve the latency of cluster change notification. Looks like we are polling for changes every 60s? This seems like a lot where for about a minute we have stale info. Is that right? Any reason not to use registry notifications for changes to llap cluster size and then compute executor counts on every ZK notification?
          hiveqa Hive QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
                Prechecks
          0 findbugs 0m 0s Findbugs executables are not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
                master Compile Tests
          +1 mvninstall 5m 46s master passed
          +1 compile 1m 4s master passed
          +1 checkstyle 0m 36s master passed
          +1 javadoc 0m 56s master passed
                Patch Compile Tests
          +1 mvninstall 1m 24s the patch passed
          +1 compile 1m 5s the patch passed
          +1 javac 1m 5s the patch passed
          +1 checkstyle 0m 34s ql: The patch generated 0 new + 155 unchanged - 2 fixed = 155 total (was 157)
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 javadoc 0m 56s the patch passed
                Other Tests
          +1 asflicense 0m 12s The patch does not generate ASF License warnings.
          12m 54s



          Subsystem Report/Notes
          Optional Tests asflicense javac javadoc findbugs checkstyle compile
          uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux
          Build tool maven
          Personality /data/hiveptest/working/yetus/dev-support/hive-personality.sh
          git revision master / 07fe7e2
          Default Java 1.8.0_111
          whitespace http://104.198.109.242/logs//PreCommit-HIVE-Build-8040/yetus/whitespace-eol.txt
          modules C: ql U: ql
          Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-8040/yetus.txt
          Powered by Apache Yetus http://yetus.apache.org

          This message was automatically generated.

          hiveqa Hive QA added a comment - -1 overall Vote Subsystem Runtime Comment       Prechecks 0 findbugs 0m 0s Findbugs executables are not available. +1 @author 0m 0s The patch does not contain any @author tags.       master Compile Tests +1 mvninstall 5m 46s master passed +1 compile 1m 4s master passed +1 checkstyle 0m 36s master passed +1 javadoc 0m 56s master passed       Patch Compile Tests +1 mvninstall 1m 24s the patch passed +1 compile 1m 5s the patch passed +1 javac 1m 5s the patch passed +1 checkstyle 0m 34s ql: The patch generated 0 new + 155 unchanged - 2 fixed = 155 total (was 157) -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 javadoc 0m 56s the patch passed       Other Tests +1 asflicense 0m 12s The patch does not generate ASF License warnings. 12m 54s Subsystem Report/Notes Optional Tests asflicense javac javadoc findbugs checkstyle compile uname Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux Build tool maven Personality /data/hiveptest/working/yetus/dev-support/hive-personality.sh git revision master / 07fe7e2 Default Java 1.8.0_111 whitespace http://104.198.109.242/logs//PreCommit-HIVE-Build-8040/yetus/whitespace-eol.txt modules C: ql U: ql Console output http://104.198.109.242/logs//PreCommit-HIVE-Build-8040/yetus.txt Powered by Apache Yetus http://yetus.apache.org This message was automatically generated.
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12899721/HIVE-17905.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 10 failed/errored test(s), 11414 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_varchar] (batchId=11)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=154)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=160)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=157)
          org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=224)
          org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
          org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
          org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8040/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8040/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8040/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.YetusPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 10 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12899721 - PreCommit-HIVE-Build

          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12899721/HIVE-17905.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 10 failed/errored test(s), 11414 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_varchar] (batchId=11) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=157) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=224) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8040/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8040/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8040/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed This message is automatically generated. ATTACHMENT ID: 12899721 - PreCommit-HIVE-Build

          Committing as is for now; we don't expect the cluster to change frequently.
          Will file a follow up jira to change the StatusForCompile to callback based model

          sershe Sergey Shelukhin added a comment - Committing as is for now; we don't expect the cluster to change frequently. Will file a follow up jira to change the StatusForCompile to callback based model
          sershe Sergey Shelukhin made changes -
          Link This issue relates to HIVE-18181 [ HIVE-18181 ]

          Committed to master. Thanks for the review!

          sershe Sergey Shelukhin added a comment - Committed to master. Thanks for the review!
          sershe Sergey Shelukhin made changes -
          Fix Version/s 3.0.0 [ 12340268 ]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]

          This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

          ashutoshc Ashutosh Chauhan added a comment - This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.
          ashutoshc Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          dfoulks Drew Foulks made changes -
          Workflow no-reopen-closed, patch-avail [ 13412648 ] Hive - no-reopen-closed, patch-avail [ 14142388 ]

          People

            sershe Sergey Shelukhin Assign to me
            sershe Sergey Shelukhin
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack