Details
Description
I was investigating test failure of TestIncrementalBackupWithBulkLoad from hbase against hadoop 3.1.1
hbase MapReduceBackupCopyJob$BackupDistCp would create listing file:
LOG.debug("creating input listing " + listing + " , totalRecords=" + totalRecords); cfg.set(DistCpConstants.CONF_LABEL_LISTING_FILE_PATH, listing); cfg.setLong(DistCpConstants.CONF_LABEL_TOTAL_NUMBER_OF_RECORDS, totalRecords);
For the test case, two bulk loaded hfiles are in the listing:
2018-10-13 14:09:24,123 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob$BackupDistCp(195): BackupDistCp : hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_ 2018-10-13 14:09:24,125 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob$BackupDistCp(195): BackupDistCp : hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_ 2018-10-13 14:09:24,125 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob$BackupDistCp(197): BackupDistCp execute for 2 files of 10242
Later on, CopyCommitter#concatFileChunks would throw the following exception:
2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590): job_local1795473782_0004 java.io.IOException: Inconsistent sequence file: current chunk file org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/ 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_ length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e- 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_ length = 5142 aclEntries = null, xAttrs = null} at org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276) at org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
The above warning shouldn't happen - the two bulk loaded hfiles are independent.
From the contents of the two CopyListingFileStatus instances, we can see that their isSplit() return false. Otherwise the following from toString should be logged:
if (isSplit()) { sb.append(", chunkOffset = ").append(this.getChunkOffset()); sb.append(", chunkLength = ").append(this.getChunkLength()); }
From hbase side, we can specify one bulk loaded hfile per job but that defeats the purpose of using DistCp.
Attachments
Attachments
- testIncrementalBackupWithBulkLoad-output.txt
- 1.09 MB
- Ted Yu
- HADOOP-15850.v6.patch
- 2 kB
- Ted Yu
- HADOOP-15850.v5.patch
- 2 kB
- Ted Yu
- HADOOP-15850.v4.patch
- 1 kB
- Ted Yu
- HADOOP-15850.v3.patch
- 1 kB
- Ted Yu
- HADOOP-15850.v2.patch
- 1 kB
- Ted Yu
- HADOOP-15850.branch-3.0.patch
- 2 kB
- Wei-Chiu Chuang
Issue Links
- causes
-
HADOOP-16018 DistCp won't reassemble chunks when blocks per chunk > 0
- Resolved
- is caused by
-
HADOOP-11794 Enable distcp to copy blocks in parallel
- Resolved
Activity
The quoted test output was from testIncrementalBackupWithBulkLoad-output.txt
yzhangal:
When you have chance, can you take a look ?
Maybe I missed some existing DistCp functionality.
Thanks
CopyCommitter#concatFileChunks is private.
It is not straight forward to override the method from DistCp user POV.
I wonder if the check for mismatching FileStatus should be refined this way:
diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/ma index 07eacb0..6177454 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java @@ -266,9 +266,9 @@ private void concatFileChunks(Configuration conf) throws IOException { // Two neighboring chunks have to be consecutive ones for the same // file, for them to be merged if (!srcFileStatus.getPath().equals(lastFileStatus.getPath()) || - srcFileStatus.getChunkOffset() != + lastFileStatus.isSplit() && (srcFileStatus.getChunkOffset() != (lastFileStatus.getChunkOffset() + - lastFileStatus.getChunkLength())) { + lastFileStatus.getChunkLength()))) { String emsg = "Inconsistent sequence file: current " + "chunk file " + srcFileStatus + " doesnt match prior " + "entry " + lastFileStatus;
The additional clause checks that lastFileStatus represents a split.
stevel@apache.org yzhangal weichiu
What do you think ?
Ted, if I understand it correctly (I have limited experience with distcp and hbase backup)
You're hitting a bug hidden in HADOOP-11794. You don't seem to be using -blocksperchunk feature, but the implementation thinks you do.
For the context, HADOOP-11794 allows distcp to copy a source file to multiple chunked intermediate files, and then "stitch" them together to the final file. This is a useful feature for storage systems that have long latency but high throughput.
Is it reproducible for HBase on Hadoop 3.1.1?
Running the backup test against hadoop 3.0.x / 3.1.y , this is easily reproducible.
I was aware of HADOOP-11794 and wondering why the per chunk feature kicks in.
weichiu:
See the link to MapReduceBackupCopyJob.java in my first comment .
We invoke DistCp programmatically.
The DistCpOptions instance for the DistCp session is not passed to CopyCommitter.
If we have per chunk information, concatFileChunks call should depend on its value.
I tried to add '-blocksperchunk 0' option when invoking DistCp:
2018-10-17 02:33:53,708 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob(416): New DistCp options: [-async, -blocksperchunk, 0, hdfs://localhost:34344/user/hbase/test-data/78931012-3303-fc71-e289-5a9726f1bfcc/data/default/test-1539743586635/2e17accd93f78be97c0f585e68f283d6/f/46480cbed054406c9ef52ff123729938_SeqId_205_, hdfs://localhost:34344/user/hbase/test-data/78931012-3303-fc71-e289-5a9726f1bfcc/data/default/test-1539743586635/2e17accd93f78be97c0f585e68f283d6/f/7e3cc96eb3f7447cb4f925df947d1fa3_SeqId_205_, hdfs://localhost:34344/backupUT/backup_1539743624592]
Still encountered 'Inconsistent sequence file' error.
Distcp code is pretty complex these days, I wouldn't be surprised if problems like this exist
In order to retrieve the per chunk information in CopyCommitter ctor,
how about using the config key DistCpOptionSwitch.BLOCKS_PER_CHUNK.getConfigLabel() ?
CopyCommitter can selectively skip concatenation when the value for the config is 0.
I tested the patch v2 in two ways:
- when there is no "-blocksperchunk" option specified, TestIncrementalBackupWithBulkLoad passes
- when positive value for "-blocksperchunk" option is specified, concatFileChunks is called - resulting in previously reported error.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 21s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
trunk Compile Tests | |||
+1 | mvninstall | 19m 8s | trunk passed |
+1 | compile | 0m 25s | trunk passed |
+1 | checkstyle | 0m 18s | trunk passed |
+1 | mvnsite | 0m 29s | trunk passed |
+1 | shadedclient | 11m 59s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 33s | trunk passed |
+1 | javadoc | 0m 20s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 27s | the patch passed |
+1 | compile | 0m 22s | the patch passed |
+1 | javac | 0m 22s | the patch passed |
-0 | checkstyle | 0m 14s | hadoop-tools/hadoop-distcp: The patch generated 2 new + 29 unchanged - 0 fixed = 31 total (was 29) |
+1 | mvnsite | 0m 24s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 12m 45s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 38s | the patch passed |
+1 | javadoc | 0m 18s | the patch passed |
Other Tests | |||
+1 | unit | 11m 41s | hadoop-distcp in the patch passed. |
+1 | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
61m 8s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12944356/HADOOP-15850.v2.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 1837f39890d6 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / d54f559 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/testReport/ |
Max. process+thread count | 339 (vs. ulimit of 10000) |
modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 20s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
trunk Compile Tests | |||
+1 | mvninstall | 19m 11s | trunk passed |
+1 | compile | 0m 27s | trunk passed |
+1 | checkstyle | 0m 19s | trunk passed |
+1 | mvnsite | 0m 29s | trunk passed |
+1 | shadedclient | 13m 20s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 45s | trunk passed |
+1 | javadoc | 0m 24s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 27s | the patch passed |
+1 | compile | 0m 22s | the patch passed |
+1 | javac | 0m 22s | the patch passed |
-0 | checkstyle | 0m 14s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29) |
+1 | mvnsite | 0m 24s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 12m 41s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 37s | the patch passed |
+1 | javadoc | 0m 19s | the patch passed |
Other Tests | |||
+1 | unit | 13m 3s | hadoop-distcp in the patch passed. |
+1 | asflicense | 0m 24s | The patch does not generate ASF License warnings. |
64m 12s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12944366/HADOOP-15850.v3.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 00891bf59a6b 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / d54f559 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/testReport/ |
Max. process+thread count | 317 (vs. ulimit of 10000) |
modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
seems reasonable to me, but others who know more of DistCP need to look at it too.
one nit: CopyCommitter now uses SLF4J, so the log statement should be
LOG.debug("blocks per chunk {}", blocksPerChunk)
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 21s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
trunk Compile Tests | |||
+1 | mvninstall | 20m 59s | trunk passed |
+1 | compile | 0m 26s | trunk passed |
+1 | checkstyle | 0m 18s | trunk passed |
+1 | mvnsite | 0m 28s | trunk passed |
+1 | shadedclient | 12m 11s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 33s | trunk passed |
+1 | javadoc | 0m 19s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 27s | the patch passed |
+1 | compile | 0m 22s | the patch passed |
+1 | javac | 0m 22s | the patch passed |
-0 | checkstyle | 0m 13s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29) |
+1 | mvnsite | 0m 23s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 12m 54s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 38s | the patch passed |
+1 | javadoc | 0m 17s | the patch passed |
Other Tests | |||
+1 | unit | 12m 3s | hadoop-distcp in the patch passed. |
+1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
63m 36s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12944559/HADOOP-15850.v4.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux cdcdbb6b9968 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 2202e00 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/testReport/ |
Max. process+thread count | 340 (vs. ulimit of 10000) |
modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
I am hoping to review the patch later today, because I'm really interested in it. stevel@apache.org please wait for me.
I think the fix makes sense. In addition, you should remove the following line in TestCopyCommitter in order to test the fix. (Comment out the line, the tests fail without the fix)
// Unset listing file path since the config is shared by // multiple tests, and some test doesn't set it, such as // testNoCommitAction, but the distcp code will check it. config.set(DistCpConstants.CONF_LABEL_LISTING_FILE_PATH, "");
Thanks for the review, looks like this bug could have been discovered sooner.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 15m 43s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 19m 0s | trunk passed |
+1 | compile | 0m 28s | trunk passed |
+1 | checkstyle | 0m 20s | trunk passed |
+1 | mvnsite | 0m 30s | trunk passed |
+1 | shadedclient | 11m 33s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 31s | trunk passed |
+1 | javadoc | 0m 17s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 25s | the patch passed |
+1 | compile | 0m 20s | the patch passed |
+1 | javac | 0m 20s | the patch passed |
-0 | checkstyle | 0m 14s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 43 unchanged - 0 fixed = 44 total (was 43) |
+1 | mvnsite | 0m 22s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 40s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 40s | the patch passed |
+1 | javadoc | 0m 19s | the patch passed |
Other Tests | |||
+1 | unit | 12m 53s | hadoop-distcp in the patch passed. |
+1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
75m 56s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12944651/HADOOP-15850.v5.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 7ef2b22cdca6 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 13cc0f5 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/testReport/ |
Max. process+thread count | 443 (vs. ulimit of 10000) |
modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Looking good; one little nit from checkstyle
./hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:105: concatFileChunks(conf);: 'if' child has incorrect indentation level 8, expected level should be 6. [Indentation]
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 14m 40s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 22m 44s | trunk passed |
+1 | compile | 0m 32s | trunk passed |
+1 | checkstyle | 0m 23s | trunk passed |
+1 | mvnsite | 0m 37s | trunk passed |
+1 | shadedclient | 13m 41s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 46s | trunk passed |
+1 | javadoc | 0m 27s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 30s | the patch passed |
+1 | compile | 0m 25s | the patch passed |
+1 | javac | 0m 25s | the patch passed |
+1 | checkstyle | 0m 16s | the patch passed |
+1 | mvnsite | 0m 29s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 14m 35s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 46s | the patch passed |
+1 | javadoc | 0m 20s | the patch passed |
Other Tests | |||
+1 | unit | 15m 2s | hadoop-distcp in the patch passed. |
+1 | asflicense | 1m 12s | The patch does not generate ASF License warnings. |
88m 2s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12944725/HADOOP-15850.v6.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 88e430ae2252 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 9bd1832 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15398/testReport/ |
Max. process+thread count | 335 (vs. ulimit of 10000) |
modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15398/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Pushed v6 patch to trunk, branch-3.2, branch-3.1, branch-3.0. Thanks yuzhihong@gmail.com for the patch and stevel@apache.org for the review
The branch-3.0 backport doesn't compile. Reverted for now.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-distcp: Compilation failure
[ERROR] /Users/weichiu/sandbox/upstream/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:[85,38] incompatible types: int cannot be converted to java.lang.Throwable
Made a trivial change to the cherry-pick for branch-3.0. Pushed the commit and posted the patch for posterity.
This seems more severe than "Major". Am I correct that this impacts downstream users of DistCp beyond HBase?
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15275 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15275/)
HADOOP-15850. CopyCommitter#concatFileChunks should check that the (weichiu: rev e2cecb681e2aab8b7c5465719cac53dce407a64c)
- (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyCommitter.java
- (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
Am I correct that this impacts downstream users of DistCp beyond HBase?
Most likely yes.
This is hbase code:
https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java#L153