Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • None
    • None

    Attachments

      1. HIVE-17841.patch
        125 kB
        Sergey Shelukhin
      2. HIVE-17841.07.patch
        145 kB
        Sergey Shelukhin
      3. HIVE-17841.06.patch
        145 kB
        Sergey Shelukhin
      4. HIVE-17841.05.patch
        141 kB
        Sergey Shelukhin
      5. HIVE-17841.04.patch
        140 kB
        Sergey Shelukhin
      6. HIVE-17841.03.patch
        139 kB
        Sergey Shelukhin
      7. HIVE-17841.02.patch
        139 kB
        Sergey Shelukhin
      8. HIVE-17841.01.patch
        119 kB
        Sergey Shelukhin

      Issue Links

        Activity

          sershe Sergey Shelukhin added a comment - - edited

          This preliminary patch is based on master although it will be committed after the pools patch. I'm pretty sure tests will fail... will look tomorrow.
          This rewrites WM to use a single thread to process almost everything that can happen to a session that is in use (so, excluding out of bounds expiration). This is because there are too many things coming from multiple sources and otherwise synchronization and reasoning about all the things happening in parallel becomes a huge pain. Also makes some things like getting session from the pool, waiting for AM registry, etc. async to facilitate that.
          There are some followup jiras that would be needed - handling internal AM restart (AM registry info update), and LLAP cluster information changes (the logic in WM is simple, the information propagation needs to be added).

          cc prasanth_j

          sershe Sergey Shelukhin added a comment - - edited This preliminary patch is based on master although it will be committed after the pools patch. I'm pretty sure tests will fail... will look tomorrow. This rewrites WM to use a single thread to process almost everything that can happen to a session that is in use (so, excluding out of bounds expiration). This is because there are too many things coming from multiple sources and otherwise synchronization and reasoning about all the things happening in parallel becomes a huge pain. Also makes some things like getting session from the pool, waiting for AM registry, etc. async to facilitate that. There are some followup jiras that would be needed - handling internal AM restart (AM registry info update), and LLAP cluster information changes (the logic in WM is simple, the information propagation needs to be added). cc prasanth_j
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12892986/HIVE-17841.patch

          SUCCESS: +1 due to 2 test(s) being added or modified.

          ERROR: -1 due to 26 failed/errored test(s), 11306 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163)
          org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16] (batchId=243)
          org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94] (batchId=243)
          org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=241)
          org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16] (batchId=241)
          org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94] (batchId=241)
          org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testClusterFractions (batchId=279)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testDestroyAndReturn (batchId=279)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueName (batchId=279)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing (batchId=279)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReopen (batchId=279)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuse (batchId=279)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithDifferentPool (batchId=279)
          org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithQueueing (batchId=279)
          org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=221)
          org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.org.apache.hive.jdbc.TestTriggersWorkloadManager (batchId=228)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow (batchId=242)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection (batchId=242)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=242)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg (batchId=242)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth (batchId=242)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth (batchId=242)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth (batchId=242)
          org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth (batchId=242)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7386/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7386/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7386/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 26 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12892986 - PreCommit-HIVE-Build

          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12892986/HIVE-17841.patch SUCCESS: +1 due to 2 test(s) being added or modified. ERROR: -1 due to 26 failed/errored test(s), 11306 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16] (batchId=243) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94] (batchId=243) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94] (batchId=241) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testClusterFractions (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testDestroyAndReturn (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueName (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReopen (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuse (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithDifferentPool (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithQueueing (batchId=279) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=221) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.org.apache.hive.jdbc.TestTriggersWorkloadManager (batchId=228) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth (batchId=242) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7386/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7386/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7386/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 26 tests failed This message is automatically generated. ATTACHMENT ID: 12892986 - PreCommit-HIVE-Build

          Fixing the tests. I still need the new tests.

          sershe Sergey Shelukhin added a comment - Fixing the tests. I still need the new tests.
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12893627/HIVE-17841.01.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 14 failed/errored test(s), 11315 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_skew] (batchId=82)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=156)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163)
          org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204)
          org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=221)
          org.apache.hadoop.hive.ql.parse.authorization.plugin.sqlstd.TestOperation2Privilege.checkHiveOperationTypeMatch (batchId=269)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1 (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime (batchId=228)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks (batchId=228)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7454/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7454/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7454/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 14 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12893627 - PreCommit-HIVE-Build

          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12893627/HIVE-17841.01.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 14 failed/errored test(s), 11315 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_skew] (batchId=82) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=221) org.apache.hadoop.hive.ql.parse.authorization.plugin.sqlstd.TestOperation2Privilege.checkHiveOperationTypeMatch (batchId=269) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1 (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks (batchId=228) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7454/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7454/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7454/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed This message is automatically generated. ATTACHMENT ID: 12893627 - PreCommit-HIVE-Build

          Adding many more tests

          sershe Sergey Shelukhin added a comment - Adding many more tests

          left some comments. Still haven't completely the epic changes to WorkloadManager.java. Will finish it later.

          prasanth_j Prasanth Jayachandran added a comment - left some comments. Still haven't completely the epic changes to WorkloadManager.java. Will finish it later.

          Addressed the CR feedback.

          sershe Sergey Shelukhin added a comment - Addressed the CR feedback.

          Addressing the CR feedback and fixing the tests

          sershe Sergey Shelukhin added a comment - Addressing the CR feedback and fixing the tests

          +1, pending tests

          prasanth_j Prasanth Jayachandran added a comment - +1, pending tests
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12894466/HIVE-17841.04.patch

          ERROR: -1 due to build exiting with an error

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7535/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7535/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7535/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
          2017-10-28 14:37:47.986
          + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
          + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
          + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'MAVEN_OPTS=-Xmx1g '
          + MAVEN_OPTS='-Xmx1g '
          + cd /data/hiveptest/working/
          + tee /data/hiveptest/logs/PreCommit-HIVE-Build-7535/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ git = \s\v\n ]]
          + [[ git = \g\i\t ]]
          + [[ -z master ]]
          + [[ -d apache-github-source-source ]]
          + [[ ! -d apache-github-source-source/.git ]]
          + [[ ! -d apache-github-source-source ]]
          + date '+%Y-%m-%d %T.%3N'
          2017-10-28 14:37:47.989
          + cd apache-github-source-source
          + git fetch origin
          + git reset --hard HEAD
          HEAD is now at a9f25c0 HIVE-17778: Add support for custom counters in trigger expression (Prasanth Jayachandran reviewed by Sergey Shelukhin)
          + git clean -f -d
          Removing ${project.basedir}/
          Removing hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java
          Removing standalone-metastore/src/gen/org/
          + git checkout master
          Already on 'master'
          Your branch is up-to-date with 'origin/master'.
          + git reset --hard origin/master
          HEAD is now at a9f25c0 HIVE-17778: Add support for custom counters in trigger expression (Prasanth Jayachandran reviewed by Sergey Shelukhin)
          + git merge --ff-only origin/master
          Already up-to-date.
          + date '+%Y-%m-%d %T.%3N'
          2017-10-28 14:37:49.360
          + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hiveptest/working/scratch/build.patch
          + [[ -f /data/hiveptest/working/scratch/build.patch ]]
          + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
          + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
          error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java:69
          error: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java: patch does not apply
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12894466 - PreCommit-HIVE-Build

          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12894466/HIVE-17841.04.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7535/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7535/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7535/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-10-28 14:37:47.986 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-7535/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-10-28 14:37:47.989 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at a9f25c0 HIVE-17778: Add support for custom counters in trigger expression (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git clean -f -d Removing ${project.basedir}/ Removing hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java Removing standalone-metastore/src/gen/org/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at a9f25c0 HIVE-17778: Add support for custom counters in trigger expression (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-10-28 14:37:49.360 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java:69 error: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12894466 - PreCommit-HIVE-Build

          Rebased on top of some recent changes on master

          sershe Sergey Shelukhin added a comment - Rebased on top of some recent changes on master
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12894847/HIVE-17841.05.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 21 failed/errored test(s), 11346 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47)
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=62)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=155)
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=101)
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=93)
          org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=205)
          org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=222)
          org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1 (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime (batchId=229)
          org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks (batchId=229)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7563/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7563/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7563/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 21 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12894847 - PreCommit-HIVE-Build

          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12894847/HIVE-17841.05.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 21 failed/errored test(s), 11346 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=62) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=155) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=101) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=93) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=205) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=222) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1 (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks (batchId=229) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7563/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7563/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7563/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed This message is automatically generated. ATTACHMENT ID: 12894847 - PreCommit-HIVE-Build

          Fixing the trigger test (at least some of them, they take forever to run so I will look at failures once it finishes). For now it will only work for one pool.
          This also hides the internals, requiring the test to go thru normal interface.
          cc prasanth_j

          sershe Sergey Shelukhin added a comment - Fixing the trigger test (at least some of them, they take forever to run so I will look at failures once it finishes). For now it will only work for one pool. This also hides the internals, requiring the test to go thru normal interface. cc prasanth_j

          Some tests still fail.. looking.

          sershe Sergey Shelukhin added a comment - Some tests still fail.. looking.

          Fixing the rest

          sershe Sergey Shelukhin added a comment - Fixing the rest

          The same patch again... HiveQA won't run

          sershe Sergey Shelukhin added a comment - The same patch again... HiveQA won't run

          Tests passed in HIVE-17907 for a combined patch that includes this patch. HiveQA is tiresome and this is blocking work by multiple people, plus the tests basically passed before, so I'm going to commit this soon.

          sershe Sergey Shelukhin added a comment - Tests passed in HIVE-17907 for a combined patch that includes this patch. HiveQA is tiresome and this is blocking work by multiple people, plus the tests basically passed before, so I'm going to commit this soon.

          Committed to master. Thanks for the review!

          sershe Sergey Shelukhin added a comment - Committed to master. Thanks for the review!

          sershe it looks like your commit have broken some tests; namely: -Dtest=TestWorkloadManager#testReuse

          kgyrtkirk Zoltan Haindrich added a comment - sershe it looks like your commit have broken some tests; namely: -Dtest=TestWorkloadManager#testReuse

          Thnx, fixed that

          sershe Sergey Shelukhin added a comment - Thnx, fixed that
          leftyl Lefty Leverenz added a comment - - edited

          Doc note: This adds hive.server2.tez.wm.worker.threads to HiveConf.java, so it needs to be documented in the wiki.

          It belongs in the HiveServer2 section of Configuration Properties, but should also be listed at the beginning of the Tez section.

          Added a TODOC3.0 label.

          Update 01/01/18: HIVE-18003 renames this parameter hive.server2.wm.worker.threads with the same default value and description, also for release 3.0.0. (Removing the TODOC3.0 label.)

          leftyl Lefty Leverenz added a comment - - edited Doc note: This adds hive.server2.tez.wm.worker.threads to HiveConf.java, so it needs to be documented in the wiki. It belongs in the HiveServer2 section of Configuration Properties, but should also be listed at the beginning of the Tez section. Configuration Properties – HiveServer2 Configuration Properties – Tez Added a TODOC3.0 label. Update 01/01/18: HIVE-18003 renames this parameter hive.server2.wm.worker.threads with the same default value and description, also for release 3.0.0. (Removing the TODOC3.0 label.)

          This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

          ashutoshc Ashutosh Chauhan added a comment - This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.

          People

            sershe Sergey Shelukhin
            sershe Sergey Shelukhin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: