Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
3.1.0, 4.0.0
Description
Support truncate acid table replication.
1. Write id allocation needs to be removed
Attachments
Attachments
- HIVE-20293.02.patch
- 92 kB
- mahesh kumar behera
- HIVE-20293.01.patch
- 92 kB
- mahesh kumar behera
Issue Links
- is related to
-
HIVE-20365 Fix warnings when regenerating thrift code
- Resolved
- links to
Activity
GitHub user maheshk114 opened a pull request:
https://github.com/apache/hive/pull/427
HIVE-20293 : Support Replication of ACID table truncate operation
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maheshk114/hive HIVE-20293
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/427.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #427
commit 334b7ddbae614a789e4e755c22ad379450dce9e1
Author: Mahesh Kumar Behera <mbehera@...>
Date: 2018-08-29T05:47:19Z
HIVE-20293 : Support Replication of ACID table truncate operation
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
master Compile Tests | |||
0 | mvndep | 0m 35s | Maven dependency ordering for branch |
+1 | mvninstall | 8m 13s | master passed |
+1 | compile | 3m 31s | master passed |
+1 | checkstyle | 1m 43s | master passed |
0 | findbugs | 2m 26s | standalone-metastore/metastore-common in master has 9 extant Findbugs warnings. |
0 | findbugs | 0m 29s | hcatalog/server-extensions in master has 3 extant Findbugs warnings. |
0 | findbugs | 0m 29s | hcatalog/webhcat/java-client in master has 3 extant Findbugs warnings. |
0 | findbugs | 0m 42s | itests/hive-unit in master has 2 extant Findbugs warnings. |
0 | findbugs | 4m 6s | ql in master has 2311 extant Findbugs warnings. |
-1 | findbugs | 0m 14s | metastore-server in master failed. |
+1 | javadoc | 3m 6s | master passed |
Patch Compile Tests | |||
0 | mvndep | 0m 9s | Maven dependency ordering for patch |
-1 | mvninstall | 0m 19s | server-extensions in the patch failed. |
-1 | mvninstall | 0m 20s | java-client in the patch failed. |
-1 | mvninstall | 0m 48s | ql in the patch failed. |
-1 | compile | 0m 19s | java-client in the patch failed. |
-1 | javac | 0m 19s | java-client in the patch failed. |
+1 | checkstyle | 0m 7s | The patch metastore-common passed checkstyle |
-1 | checkstyle | 0m 12s | hcatalog/server-extensions: The patch generated 3 new + 76 unchanged - 4 fixed = 79 total (was 80) |
+1 | checkstyle | 0m 12s | hcatalog/webhcat/java-client: The patch generated 0 new + 56 unchanged - 2 fixed = 56 total (was 58) |
-1 | checkstyle | 0m 19s | itests/hive-unit: The patch generated 2 new + 81 unchanged - 0 fixed = 83 total (was 81) |
-1 | checkstyle | 0m 46s | ql: The patch generated 1 new + 626 unchanged - 0 fixed = 627 total (was 626) |
+1 | checkstyle | 0m 8s | The patch metastore-server passed checkstyle |
-1 | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
-1 | findbugs | 0m 21s | java-client in the patch failed. |
-1 | findbugs | 4m 17s | ql generated 1 new + 2310 unchanged - 1 fixed = 2311 total (was 2311) |
-1 | findbugs | 0m 14s | metastore-server in the patch failed. |
+1 | javadoc | 3m 9s | the patch passed |
Other Tests | |||
+1 | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
47m 54s |
Reason | Tests |
---|---|
FindBugs | module:ql |
Nullcheck of replicationSpec at line 228 of value previously dereferenced in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(boolean, boolean, boolean, boolean, boolean, String, String, String, LinkedHashMap, String, EximUtil$SemanticAnalyzerWrapperContext, UpdatedMetaDataTracker, HiveTxnManager, long) At ImportSemanticAnalyzer.java:228 of value previously dereferenced in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(boolean, boolean, boolean, boolean, boolean, String, String, String, LinkedHashMap, String, EximUtil$SemanticAnalyzerWrapperContext, UpdatedMetaDataTracker, HiveTxnManager, long) At ImportSemanticAnalyzer.java:[line 228] |
This message was automatically generated.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12937777/HIVE-20293.01.patch
SUCCESS: +1 due to 2 test(s) being added or modified.
ERROR: -1 due to 2 failed/errored test(s), 14916 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile] (batchId=2) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testViewsReplication (batchId=241)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13532/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13532/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13532/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed
This message is automatically generated.
ATTACHMENT ID: 12937777 - PreCommit-HIVE-Build
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
master Compile Tests | |||
0 | mvndep | 0m 36s | Maven dependency ordering for branch |
+1 | mvninstall | 7m 49s | master passed |
+1 | compile | 3m 28s | master passed |
+1 | checkstyle | 1m 43s | master passed |
0 | findbugs | 2m 23s | standalone-metastore/metastore-common in master has 9 extant Findbugs warnings. |
0 | findbugs | 0m 28s | hcatalog/server-extensions in master has 3 extant Findbugs warnings. |
0 | findbugs | 0m 29s | hcatalog/webhcat/java-client in master has 3 extant Findbugs warnings. |
0 | findbugs | 0m 41s | itests/hive-unit in master has 2 extant Findbugs warnings. |
0 | findbugs | 4m 3s | ql in master has 2310 extant Findbugs warnings. |
-1 | findbugs | 0m 14s | metastore-server in master failed. |
+1 | javadoc | 3m 4s | master passed |
Patch Compile Tests | |||
0 | mvndep | 0m 8s | Maven dependency ordering for patch |
-1 | mvninstall | 0m 18s | server-extensions in the patch failed. |
-1 | mvninstall | 0m 20s | java-client in the patch failed. |
-1 | mvninstall | 0m 50s | ql in the patch failed. |
-1 | compile | 0m 21s | java-client in the patch failed. |
-1 | javac | 0m 21s | java-client in the patch failed. |
+1 | checkstyle | 0m 7s | The patch metastore-common passed checkstyle |
-1 | checkstyle | 0m 12s | hcatalog/server-extensions: The patch generated 3 new + 76 unchanged - 4 fixed = 79 total (was 80) |
+1 | checkstyle | 0m 12s | hcatalog/webhcat/java-client: The patch generated 0 new + 56 unchanged - 2 fixed = 56 total (was 58) |
-1 | checkstyle | 0m 17s | itests/hive-unit: The patch generated 2 new + 81 unchanged - 0 fixed = 83 total (was 81) |
-1 | checkstyle | 0m 45s | ql: The patch generated 2 new + 626 unchanged - 0 fixed = 628 total (was 626) |
+1 | checkstyle | 0m 7s | The patch metastore-server passed checkstyle |
-1 | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
-1 | findbugs | 0m 19s | java-client in the patch failed. |
-1 | findbugs | 4m 19s | ql generated 1 new + 2309 unchanged - 1 fixed = 2310 total (was 2310) |
-1 | findbugs | 0m 14s | metastore-server in the patch failed. |
+1 | javadoc | 3m 9s | the patch passed |
Other Tests | |||
+1 | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
47m 8s |
Reason | Tests |
---|---|
FindBugs | module:ql |
Nullcheck of replicationSpec at line 228 of value previously dereferenced in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(boolean, boolean, boolean, boolean, boolean, String, String, String, LinkedHashMap, String, EximUtil$SemanticAnalyzerWrapperContext, UpdatedMetaDataTracker, HiveTxnManager, long) At ImportSemanticAnalyzer.java:228 of value previously dereferenced in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(boolean, boolean, boolean, boolean, boolean, String, String, String, LinkedHashMap, String, EximUtil$SemanticAnalyzerWrapperContext, UpdatedMetaDataTracker, HiveTxnManager, long) At ImportSemanticAnalyzer.java:[line 228] |
This message was automatically generated.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12938215/HIVE-20293.02.patch
SUCCESS: +1 due to 2 test(s) being added or modified.
SUCCESS: +1 due to 14920 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13580/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13580/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13580/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase
This message is automatically generated.
ATTACHMENT ID: 12938215 - PreCommit-HIVE-Build
02.patch committed to master!
Thanks maheshk114 for the contribution!
patch not required for branch-3 as the related changes are not committed to branch-3. Even without this change , truncate table is working fine for acid tables in branch-3.
Added the write id to the alter table / alter partition messages to get the valid write id lists on the target cluster. Added a new test for alter table. The truncate table operation (removing files) is still non transactional.