Vinod Kumar Vavilapalli added a comment - 02/Jun/17 23:58 Zhe Zhang , I'm pushing for a 2.7.4, can you explain why it has to be reverted?

Akira Ajisaka added a comment - 04/Jun/17 00:08 +1 for reverting HDFS-7933 since it has changed the output format of hdfs fsck command.

Zhe Zhang added a comment - 05/Jun/17 23:22 Per latest discussion under HDFS-7933 , repurposing this JIRA to revert the incompatible fsck output (but keep the calculation logic behind decommissioning and decommissioned replicas). Akira Ajisaka , Vinod Kumar Vavilapalli LMK if the above plan sounds good.

Akira Ajisaka added a comment - 06/Jun/17 04:58 Sounds good. Thanks Zhe Zhang .

Zhe Zhang added a comment - 06/Jun/17 05:10 - edited Attaching v00 patch to revert NamenodeFsck and TestFsck .

Brahma Reddy Battula added a comment - 06/Jun/17 06:50 Zhe Zhang thanks for reporting and working on this. Should we return only decommissioned replicas(like before HDFS-7933 )..? May be you can just return decommissioned in NumberReplicas#decommissionedReplicas ..? 248 out.println( "No. of decommission Replica: " 249 + numberReplicas.decommissionedAndDecommissioning()); And as I mentioned in HDFS-7933 , we can make it configurable in seperate jira..?

Hadoop QA added a comment - 06/Jun/17 07:34 -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 38s branch-2.7 passed +1 compile 1m 7s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 6s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 26s branch-2.7 passed +1 mvnsite 1m 4s branch-2.7 passed +1 findbugs 3m 9s branch-2.7 passed +1 javadoc 1m 1s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 52s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 1m 1s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_131 +1 javac 1m 2s the patch passed +1 compile 1m 4s the patch passed with JDK v1.7.0_131 +1 javac 1m 4s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85) +1 mvnsite 1m 0s the patch passed -1 whitespace 0m 0s The patch has 1304 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 32s The patch 70 line(s) with tabs. +1 findbugs 3m 19s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_131 -1 unit 45m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 135m 52s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestReplaceDatanodeOnFailure JDK v1.7.0_131 Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11743 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871480/HDFS-11743-branch-2.7.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 08514e0aebc7 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / db81fbb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19785/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19785/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Zhe Zhang added a comment - 06/Jun/17 15:14 Thanks Brahma Reddy Battula for taking a look. Yes you are right, that output should only be decommissioned. Attaching v01 patch to fix it. I also verified that none of the reported unit test failures could be reproduced locally. As for whether we should add the HDFS-7933 fsck output as a configurable option, Akira Ajisaka Vinod Kumar Vavilapalli Konstantin Shvachko any opinions?

Hadoop QA added a comment - 06/Jun/17 17:29 -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 59s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 4s branch-2.7 passed +1 compile 0m 58s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 23s branch-2.7 passed +1 mvnsite 0m 58s branch-2.7 passed +1 findbugs 2m 54s branch-2.7 passed +1 javadoc 0m 56s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 46s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 0m 52s the patch passed +1 compile 0m 54s the patch passed with JDK v1.8.0_131 +1 javac 0m 54s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_131 +1 javac 0m 57s the patch passed -0 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85) +1 mvnsite 0m 55s the patch passed -1 whitespace 0m 0s The patch has 1646 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 44s The patch 74 line(s) with tabs. +1 findbugs 3m 8s the patch passed +1 javadoc 0m 55s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_131 -1 unit 45m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 131m 31s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11743 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871619/HDFS-11743-branch-2.7.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 335afe1e94db 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / db81fbb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19801/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19801/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Konstantin Shvachko added a comment - 06/Jun/17 18:38 whether we should add the HDFS-7933 fsck output as a configurable option I don't think we should have it configurable. This will create problems going forward to 2.8 and trunk, if they wont have the option. So just restoring the output for 2.7.4 sounds good to me.

Zhe Zhang added a comment - 06/Jun/17 21:21 Thanks for confirming Konstantin Shvachko . I verified reported unit test failures and couldn't reproduce them locally. TestDecommissioningStatus is potentially related but I couldn't reproduce the failure after running locally for 20+ times. Just triggered Jenkins again. I suspect the reason is that during the Jenkins run of the unit test, one DN in the minicluster was having issue so one more block was considered under-replicated.

Hadoop QA added a comment - 06/Jun/17 23:38 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 18s branch-2.7 passed +1 compile 1m 8s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 25s branch-2.7 passed +1 mvnsite 1m 0s branch-2.7 passed +1 findbugs 3m 0s branch-2.7 passed +1 javadoc 1m 0s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 53s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 0m 57s the patch passed +1 compile 0m 58s the patch passed with JDK v1.8.0_131 +1 javac 0m 58s the patch passed +1 compile 1m 1s the patch passed with JDK v1.7.0_131 +1 javac 1m 1s the patch passed -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85) +1 mvnsite 0m 59s the patch passed -1 whitespace 0m 0s The patch has 1953 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 47s The patch 74 line(s) with tabs. +1 findbugs 3m 8s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_131 -1 unit 54m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 139m 53s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11743 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871619/HDFS-11743-branch-2.7.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4616b8bd160 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / db81fbb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19807/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19807/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Zhe Zhang added a comment - 06/Jun/17 23:42 Brahma Reddy Battula Konstantin Shvachko Please kindly review v01 patch. I have verified all reported test failures (none could be reproduced locally). Thanks!

Akira Ajisaka added a comment - 07/Jun/17 09:35 Thanks! +1 pending Brahma Reddy Battula and Konstantin Shvachko 's reviews.

Brahma Reddy Battula added a comment - 07/Jun/17 14:02 Zhe Zhang thanks for updating the patch.. +1 on latest patch. I believe TestDecommissioningStatus failure unrelated..It will be random failure, 2 times failed when I ran whole class 6 times.Looks incomplete(rbw) blocks are not written in failure(FBR is triggered) case,will dig more.

Zhe Zhang added a comment - 07/Jun/17 16:00 Thanks Akira Ajisaka and Brahma Reddy Battula for the reviews! I just committed the patch to branch-2.7.

Xiao Chen added a comment - 27/Sep/17 23:46 Hi Zhe Zhang , Was reverting TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt intentional? It looks unrelated to HDFS-7933 and pretty contained by HDFS-11445 . I just tried to run it on latest branch-2.7, and it still passes.

Brahma Reddy Battula added a comment - 28/Sep/17 04:54 Oh,Yes. Xiao Chen thanks for finding. Looks it's unintenational.

Xiao Chen added a comment - 29/Sep/17 20:15 Thanks Brahma Reddy Battula for confirming. Here's the patch to add that test back. Ran locally and passed. If no objections, I will commit this to branch-2.7 as an addendum of HDFS-11743 next week.

Brahma Reddy Battula added a comment - 05/Oct/17 14:24 I think, we need to handle in seperate jira..? since 2.7.4 release is already given..?

Xiao Chen added a comment - 05/Oct/17 17:47 Sure, filed HDFS-12596 and provided a patch there.