Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Attachments
Attachments
- HIVE-17841.patch
- 125 kB
- Sergey Shelukhin
- HIVE-17841.07.patch
- 145 kB
- Sergey Shelukhin
- HIVE-17841.06.patch
- 145 kB
- Sergey Shelukhin
- HIVE-17841.05.patch
- 141 kB
- Sergey Shelukhin
- HIVE-17841.04.patch
- 140 kB
- Sergey Shelukhin
- HIVE-17841.03.patch
- 139 kB
- Sergey Shelukhin
- HIVE-17841.02.patch
- 139 kB
- Sergey Shelukhin
- HIVE-17841.01.patch
- 119 kB
- Sergey Shelukhin
Issue Links
- is related to
-
HIVE-18003 add explicit jdbc connection string args for mappings
- Closed
- links to
Activity
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12892986/HIVE-17841.patch
SUCCESS: +1 due to 2 test(s) being added or modified.
ERROR: -1 due to 26 failed/errored test(s), 11306 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16] (batchId=243) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94] (batchId=243) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94] (batchId=241) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testClusterFractions (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testDestroyAndReturn (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueName (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReopen (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuse (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithDifferentPool (batchId=279) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithQueueing (batchId=279) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=221) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.org.apache.hive.jdbc.TestTriggersWorkloadManager (batchId=228) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth (batchId=242) org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth (batchId=242)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7386/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7386/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7386/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 26 tests failed
This message is automatically generated.
ATTACHMENT ID: 12892986 - PreCommit-HIVE-Build
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12893627/HIVE-17841.01.patch
SUCCESS: +1 due to 1 test(s) being added or modified.
ERROR: -1 due to 14 failed/errored test(s), 11315 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_skew] (batchId=82) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=221) org.apache.hadoop.hive.ql.parse.authorization.plugin.sqlstd.TestOperation2Privilege.checkHiveOperationTypeMatch (batchId=269) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1 (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime (batchId=228) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks (batchId=228)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7454/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7454/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7454/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed
This message is automatically generated.
ATTACHMENT ID: 12893627 - PreCommit-HIVE-Build
left some comments. Still haven't completely the epic changes to WorkloadManager.java. Will finish it later.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894466/HIVE-17841.04.patch
ERROR: -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7535/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7535/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7535/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-10-28 14:37:47.986 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-7535/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-10-28 14:37:47.989 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at a9f25c0 HIVE-17778: Add support for custom counters in trigger expression (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git clean -f -d Removing ${project.basedir}/ Removing hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseMetaHook.java Removing standalone-metastore/src/gen/org/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at a9f25c0 HIVE-17778: Add support for custom counters in trigger expression (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-10-28 14:37:49.360 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java:69 error: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 '
This message is automatically generated.
ATTACHMENT ID: 12894466 - PreCommit-HIVE-Build
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12894847/HIVE-17841.05.patch
SUCCESS: +1 due to 1 test(s) being added or modified.
ERROR: -1 due to 21 failed/errored test(s), 11346 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=62) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=155) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=101) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=93) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=205) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=222) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedFiles (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1 (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2 (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerTotalTasks (batchId=229)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7563/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7563/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7563/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed
This message is automatically generated.
ATTACHMENT ID: 12894847 - PreCommit-HIVE-Build
Fixing the trigger test (at least some of them, they take forever to run so I will look at failures once it finishes). For now it will only work for one pool.
This also hides the internals, requiring the test to go thru normal interface.
cc prasanth_j
Tests passed in HIVE-17907 for a combined patch that includes this patch. HiveQA is tiresome and this is blocking work by multiple people, plus the tests basically passed before, so I'm going to commit this soon.
sershe it looks like your commit have broken some tests; namely: -Dtest=TestWorkloadManager#testReuse
Doc note: This adds hive.server2.tez.wm.worker.threads to HiveConf.java, so it needs to be documented in the wiki.
It belongs in the HiveServer2 section of Configuration Properties, but should also be listed at the beginning of the Tez section.
Added a TODOC3.0 label.
Update 01/01/18: HIVE-18003 renames this parameter hive.server2.wm.worker.threads with the same default value and description, also for release 3.0.0. (Removing the TODOC3.0 label.)
This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.
This preliminary patch is based on master although it will be committed after the pools patch. I'm pretty sure tests will fail... will look tomorrow.
This rewrites WM to use a single thread to process almost everything that can happen to a session that is in use (so, excluding out of bounds expiration). This is because there are too many things coming from multiple sources and otherwise synchronization and reasoning about all the things happening in parallel becomes a huge pain. Also makes some things like getting session from the pool, waiting for AM registry, etc. async to facilitate that.
There are some followup jiras that would be needed - handling internal AM restart (AM registry info update), and LLAP cluster information changes (the logic in WM is simple, the information propagation needs to be added).
cc prasanth_j