Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
hdfs dfsadmin -report has very useful information about the cluster. There are some existing customized tools that depend on this command functionality. We should add unit test for it. Specially,
- If one datanode is dead, the report should indicate this
- If one block is corrupt, the "Missing blocks:" field should report this
- TBD...
Attachments
Attachments
- HDFS-11083.004.patch
- 7 kB
- Xiaobing Zhou
- HDFS-11083.003.patch
- 7 kB
- Xiaobing Zhou
- HDFS-11083.002.patch
- 10 kB
- Xiaobing Zhou
- HDFS-11083.001.patch
- 11 kB
- Xiaobing Zhou
- HDFS-11083.000.patch
- 11 kB
- Xiaobing Zhou
Issue Links
- is related to
-
HDFS-11122 TestDFSAdmin#testReportCommand fails due to timed out
- Resolved
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 19s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
+1 | mvninstall | 7m 0s | trunk passed |
+1 | compile | 0m 45s | trunk passed |
+1 | checkstyle | 0m 27s | trunk passed |
+1 | mvnsite | 0m 52s | trunk passed |
+1 | mvneclipse | 0m 13s | trunk passed |
+1 | findbugs | 1m 46s | trunk passed |
+1 | javadoc | 0m 38s | trunk passed |
+1 | mvninstall | 0m 46s | the patch passed |
+1 | compile | 0m 45s | the patch passed |
+1 | javac | 0m 45s | the patch passed |
-0 | checkstyle | 0m 25s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 59 unchanged - 0 fixed = 65 total (was 59) |
+1 | mvnsite | 0m 53s | the patch passed |
+1 | mvneclipse | 0m 10s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 1m 52s | the patch passed |
+1 | javadoc | 0m 39s | the patch passed |
-1 | unit | 74m 58s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
94m 1s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
hadoop.hdfs.TestRollingUpgrade |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:e809691 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12837846/HDFS-11083.000.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 8a5f19bfeafc 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / de3b4aa |
Default Java | 1.8.0_101 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/17460/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17460/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17460/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17460/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Posted patch v002, I just noticed previous patches contain unnecessary hack-in stuff.
The patch looks good to me overall. Thanks,
- Is cluster.setDataNodeDead helpful here?
467 /* wait until DN report is updated */ 468 GenericTestUtils.waitFor(new Supplier<Boolean>() { 469 @Override 470 public Boolean get() { 471 DatanodeInfo[] nodeInfo = null; 472 try { 473 nodeInfo = client.datanodeReport(DatanodeReportType.DEAD); 474 } catch (IOException e) { 475 return false; 476 } 477 return nodeInfo != null && nodeInfo.length == 1; 478 } 479 }, 100, 60000);
- I prefer not not to change the existing order the imports as this may cause backporting conflict which is traumatic for tools.
25 import static org.hamcrest.CoreMatchers.allOf; 26 import static org.hamcrest.CoreMatchers.anyOf; 27 import static org.hamcrest.CoreMatchers.containsString; 28 import static org.hamcrest.CoreMatchers.is; 29 import static org.hamcrest.CoreMatchers.not; 30 import static org.junit.Assert.assertEquals; 31 import static org.junit.Assert.assertThat; 32 import static org.junit.Assert.assertTrue; 33 import static org.mockito.Matchers.any; 34 import static org.mockito.Mockito.mock; 35 import static org.mockito.Mockito.when;
- Are you saying "Fail to corrupt all replicas for block " + block as assertion message?
496 assertEquals("No all replicas corrupted", repl_factor, 497 blockFilesCorrupted);
- Better {{ fs.setReplication(file, fs.setReplication(file, (short)2); + 1);}}
499 /* 500 * Increase replication factor, this should invoke transfer request. 501 * Receiving datanode fails on checksum and reports it to namenode 502 */ 503 fs.setReplication(file, (short)2);
- printout() is used ever?
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 21s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
+1 | mvninstall | 7m 12s | trunk passed |
+1 | compile | 0m 46s | trunk passed |
+1 | checkstyle | 0m 28s | trunk passed |
+1 | mvnsite | 0m 54s | trunk passed |
+1 | mvneclipse | 0m 12s | trunk passed |
+1 | findbugs | 1m 44s | trunk passed |
+1 | javadoc | 0m 46s | trunk passed |
+1 | mvninstall | 0m 50s | the patch passed |
+1 | compile | 0m 46s | the patch passed |
+1 | javac | 0m 46s | the patch passed |
-0 | checkstyle | 0m 28s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 59 unchanged - 0 fixed = 61 total (was 59) |
+1 | mvnsite | 0m 52s | the patch passed |
+1 | mvneclipse | 0m 12s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 1m 49s | the patch passed |
+1 | javadoc | 0m 38s | the patch passed |
-1 | unit | 92m 43s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
112m 21s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestBlocksScheduledCounter |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency | |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
hadoop.fs.TestSymlinkHdfsFileContext |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:e809691 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12837861/HDFS-11083.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 330fbad64222 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / de3b4aa |
Default Java | 1.8.0_101 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/17461/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17461/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17461/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17461/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 4m 12s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | mvninstall | 9m 26s | trunk passed |
+1 | compile | 1m 0s | trunk passed |
+1 | checkstyle | 0m 30s | trunk passed |
+1 | mvnsite | 1m 5s | trunk passed |
+1 | mvneclipse | 0m 14s | trunk passed |
+1 | findbugs | 2m 8s | trunk passed |
+1 | javadoc | 0m 46s | trunk passed |
+1 | mvninstall | 0m 55s | the patch passed |
+1 | compile | 0m 49s | the patch passed |
+1 | javac | 0m 49s | the patch passed |
-0 | checkstyle | 0m 26s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
+1 | mvnsite | 1m 3s | the patch passed |
+1 | mvneclipse | 0m 13s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 2m 8s | the patch passed |
+1 | javadoc | 0m 44s | the patch passed |
-1 | unit | 63m 1s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
90m 32s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:e809691 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12837871/HDFS-11083.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 1072b825a0a8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 3dbad5d |
Default Java | 1.8.0_101 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/17462/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17462/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17462/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17462/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks liuml07. I posted patch v003. It addressed the 5 points though I didn't quite get #4.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | mvninstall | 7m 2s | trunk passed |
+1 | compile | 0m 46s | trunk passed |
+1 | checkstyle | 0m 27s | trunk passed |
+1 | mvnsite | 0m 53s | trunk passed |
+1 | mvneclipse | 0m 13s | trunk passed |
+1 | findbugs | 1m 44s | trunk passed |
+1 | javadoc | 0m 40s | trunk passed |
+1 | mvninstall | 0m 47s | the patch passed |
+1 | compile | 0m 42s | the patch passed |
+1 | javac | 0m 42s | the patch passed |
-0 | checkstyle | 0m 23s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
+1 | mvnsite | 0m 49s | the patch passed |
+1 | mvneclipse | 0m 10s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 1m 48s | the patch passed |
+1 | javadoc | 0m 37s | the patch passed |
-1 | unit | 74m 18s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
93m 13s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:e809691 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838011/HDFS-11083.003.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 1cad489ee95e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / dbb133c |
Default Java | 1.8.0_101 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/17473/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17473/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17473/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
537 fs.setReplication(file, (short) (replFactor + 1));
The replFactor is short type, isn't it?
It addressed the 5 points though I didn't quite get #4.
That was a bad format, I think you got the idea (using replFactor +1 instead of 2), as the above comment.
+1
Posted v004 to fix the check style.
short + 1 is type of int, so needs explicit cast to avoid warning, thanks though.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 26s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | mvninstall | 8m 0s | trunk passed |
+1 | compile | 0m 46s | trunk passed |
+1 | checkstyle | 0m 26s | trunk passed |
+1 | mvnsite | 0m 52s | trunk passed |
+1 | mvneclipse | 0m 12s | trunk passed |
+1 | findbugs | 1m 44s | trunk passed |
+1 | javadoc | 0m 39s | trunk passed |
+1 | mvninstall | 0m 46s | the patch passed |
+1 | compile | 0m 42s | the patch passed |
+1 | javac | 0m 42s | the patch passed |
+1 | checkstyle | 0m 22s | the patch passed |
+1 | mvnsite | 0m 50s | the patch passed |
+1 | mvneclipse | 0m 9s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 1m 46s | the patch passed |
+1 | javadoc | 0m 37s | the patch passed |
+1 | unit | 60m 52s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
80m 41s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:e809691 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838072/HDFS-11083.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 741218d5fced 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 29e3b34 |
Default Java | 1.8.0_101 |
findbugs | v3.0.0 |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17477/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17477/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Committed to trunk through branch-2.8 branches. I resolved minor conflicts when committing. Thanks for your contribution, xiaobingo.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10795 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10795/)
HDFS-11083. Add unit test for DFSAdmin -report command. Contributed by (liuml07: rev 62d8c17dfda75a6a6de06aedad2f22699a1cbad6)
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
Posted initial patch for reviews.