Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11743

Revert the incompatible fsck reporting output in HDFS-7933 from branch-2.7

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.4
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. HDFS-11743-branch-2.7.00.patch
      11 kB
      Zhe Zhang
    2. HDFS-11743-branch-2.7.01.patch
      11 kB
      Zhe Zhang
    3. HDFS-11743.addendum.patch
      2 kB
      Xiao Chen

      Issue Links

        Activity

        Hide
        zhz Zhe Zhang added a comment -

        Thanks Xiao Chen and Brahma Reddy Battula for the discussion (and sorry I missed it). Correct, the testFsckCorruptWhenOneReplicaIsCorrupt change was my mistake.

        Show
        zhz Zhe Zhang added a comment - Thanks Xiao Chen and Brahma Reddy Battula for the discussion (and sorry I missed it). Correct, the testFsckCorruptWhenOneReplicaIsCorrupt change was my mistake.
        Hide
        xiaochen Xiao Chen added a comment -

        Sure, filed HDFS-12596 and provided a patch there.

        Show
        xiaochen Xiao Chen added a comment - Sure, filed HDFS-12596 and provided a patch there.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        I think, we need to handle in seperate jira..? since 2.7.4 release is already given..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - I think, we need to handle in seperate jira..? since 2.7.4 release is already given..?
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Brahma Reddy Battula for confirming.

        Here's the patch to add that test back. Ran locally and passed.

        If no objections, I will commit this to branch-2.7 as an addendum of HDFS-11743 next week.

        Show
        xiaochen Xiao Chen added a comment - Thanks Brahma Reddy Battula for confirming. Here's the patch to add that test back. Ran locally and passed. If no objections, I will commit this to branch-2.7 as an addendum of HDFS-11743 next week.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Oh,Yes.Xiao Chen thanks for finding. Looks it's unintenational.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Oh,Yes. Xiao Chen thanks for finding. Looks it's unintenational.
        Hide
        xiaochen Xiao Chen added a comment -

        Hi Zhe Zhang,
        Was reverting TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt intentional? It looks unrelated to HDFS-7933 and pretty contained by HDFS-11445. I just tried to run it on latest branch-2.7, and it still passes.

        Show
        xiaochen Xiao Chen added a comment - Hi Zhe Zhang , Was reverting TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt intentional? It looks unrelated to HDFS-7933 and pretty contained by HDFS-11445 . I just tried to run it on latest branch-2.7, and it still passes.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Akira Ajisaka and Brahma Reddy Battula for the reviews! I just committed the patch to branch-2.7.

        Show
        zhz Zhe Zhang added a comment - Thanks Akira Ajisaka and Brahma Reddy Battula for the reviews! I just committed the patch to branch-2.7.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Zhe Zhang thanks for updating the patch.. +1 on latest patch.

        I believe TestDecommissioningStatus failure unrelated..It will be random failure, 2 times failed when I ran whole class 6 times.Looks incomplete(rbw) blocks are not written in failure(FBR is triggered) case,will dig more.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Zhe Zhang thanks for updating the patch.. +1 on latest patch. I believe TestDecommissioningStatus failure unrelated..It will be random failure, 2 times failed when I ran whole class 6 times.Looks incomplete(rbw) blocks are not written in failure(FBR is triggered) case,will dig more.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Thanks! +1 pending Brahma Reddy Battula and Konstantin Shvachko's reviews.

        Show
        ajisakaa Akira Ajisaka added a comment - Thanks! +1 pending Brahma Reddy Battula and Konstantin Shvachko 's reviews.
        Hide
        zhz Zhe Zhang added a comment -

        Brahma Reddy Battula Konstantin Shvachko Please kindly review v01 patch. I have verified all reported test failures (none could be reproduced locally). Thanks!

        Show
        zhz Zhe Zhang added a comment - Brahma Reddy Battula Konstantin Shvachko Please kindly review v01 patch. I have verified all reported test failures (none could be reproduced locally). Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 27s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 18s branch-2.7 passed
        +1 compile 1m 8s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_131
        +1 checkstyle 0m 25s branch-2.7 passed
        +1 mvnsite 1m 0s branch-2.7 passed
        +1 findbugs 3m 0s branch-2.7 passed
        +1 javadoc 1m 0s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 53s branch-2.7 passed with JDK v1.7.0_131
        +1 mvninstall 0m 57s the patch passed
        +1 compile 0m 58s the patch passed with JDK v1.8.0_131
        +1 javac 0m 58s the patch passed
        +1 compile 1m 1s the patch passed with JDK v1.7.0_131
        +1 javac 1m 1s the patch passed
        -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85)
        +1 mvnsite 0m 59s the patch passed
        -1 whitespace 0m 0s The patch has 1953 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 47s The patch 74 line(s) with tabs.
        +1 findbugs 3m 8s the patch passed
        +1 javadoc 0m 57s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 47s the patch passed with JDK v1.7.0_131
        -1 unit 54m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        139m 53s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:67e87c9
        JIRA Issue HDFS-11743
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871619/HDFS-11743-branch-2.7.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c4616b8bd160 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / db81fbb
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19807/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19807/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 18s branch-2.7 passed +1 compile 1m 8s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 25s branch-2.7 passed +1 mvnsite 1m 0s branch-2.7 passed +1 findbugs 3m 0s branch-2.7 passed +1 javadoc 1m 0s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 53s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 0m 57s the patch passed +1 compile 0m 58s the patch passed with JDK v1.8.0_131 +1 javac 0m 58s the patch passed +1 compile 1m 1s the patch passed with JDK v1.7.0_131 +1 javac 1m 1s the patch passed -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85) +1 mvnsite 0m 59s the patch passed -1 whitespace 0m 0s The patch has 1953 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 47s The patch 74 line(s) with tabs. +1 findbugs 3m 8s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_131 -1 unit 54m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 139m 53s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11743 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871619/HDFS-11743-branch-2.7.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4616b8bd160 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / db81fbb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19807/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19807/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19807/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for confirming Konstantin Shvachko.

        I verified reported unit test failures and couldn't reproduce them locally. TestDecommissioningStatus is potentially related but I couldn't reproduce the failure after running locally for 20+ times. Just triggered Jenkins again. I suspect the reason is that during the Jenkins run of the unit test, one DN in the minicluster was having issue so one more block was considered under-replicated.

        Show
        zhz Zhe Zhang added a comment - Thanks for confirming Konstantin Shvachko . I verified reported unit test failures and couldn't reproduce them locally. TestDecommissioningStatus is potentially related but I couldn't reproduce the failure after running locally for 20+ times. Just triggered Jenkins again. I suspect the reason is that during the Jenkins run of the unit test, one DN in the minicluster was having issue so one more block was considered under-replicated.
        Hide
        shv Konstantin Shvachko added a comment -

        whether we should add the HDFS-7933 fsck output as a configurable option

        I don't think we should have it configurable. This will create problems going forward to 2.8 and trunk, if they wont have the option. So just restoring the output for 2.7.4 sounds good to me.

        Show
        shv Konstantin Shvachko added a comment - whether we should add the HDFS-7933 fsck output as a configurable option I don't think we should have it configurable. This will create problems going forward to 2.8 and trunk, if they wont have the option. So just restoring the output for 2.7.4 sounds good to me.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 10m 59s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 4s branch-2.7 passed
        +1 compile 0m 58s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_131
        +1 checkstyle 0m 23s branch-2.7 passed
        +1 mvnsite 0m 58s branch-2.7 passed
        +1 findbugs 2m 54s branch-2.7 passed
        +1 javadoc 0m 56s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 46s branch-2.7 passed with JDK v1.7.0_131
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 54s the patch passed with JDK v1.8.0_131
        +1 javac 0m 54s the patch passed
        +1 compile 0m 57s the patch passed with JDK v1.7.0_131
        +1 javac 0m 57s the patch passed
        -0 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85)
        +1 mvnsite 0m 55s the patch passed
        -1 whitespace 0m 0s The patch has 1646 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 44s The patch 74 line(s) with tabs.
        +1 findbugs 3m 8s the patch passed
        +1 javadoc 0m 55s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 42s the patch passed with JDK v1.7.0_131
        -1 unit 45m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        131m 31s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:67e87c9
        JIRA Issue HDFS-11743
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871619/HDFS-11743-branch-2.7.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 335afe1e94db 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / db81fbb
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19801/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19801/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 59s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 4s branch-2.7 passed +1 compile 0m 58s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 23s branch-2.7 passed +1 mvnsite 0m 58s branch-2.7 passed +1 findbugs 2m 54s branch-2.7 passed +1 javadoc 0m 56s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 46s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 0m 52s the patch passed +1 compile 0m 54s the patch passed with JDK v1.8.0_131 +1 javac 0m 54s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_131 +1 javac 0m 57s the patch passed -0 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85) +1 mvnsite 0m 55s the patch passed -1 whitespace 0m 0s The patch has 1646 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 44s The patch 74 line(s) with tabs. +1 findbugs 3m 8s the patch passed +1 javadoc 0m 55s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_131 -1 unit 45m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 131m 31s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11743 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871619/HDFS-11743-branch-2.7.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 335afe1e94db 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / db81fbb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19801/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19801/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19801/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Brahma Reddy Battula for taking a look. Yes you are right, that output should only be decommissioned. Attaching v01 patch to fix it.

        I also verified that none of the reported unit test failures could be reproduced locally.

        As for whether we should add the HDFS-7933 fsck output as a configurable option, Akira Ajisaka Vinod Kumar Vavilapalli Konstantin Shvachko any opinions?

        Show
        zhz Zhe Zhang added a comment - Thanks Brahma Reddy Battula for taking a look. Yes you are right, that output should only be decommissioned. Attaching v01 patch to fix it. I also verified that none of the reported unit test failures could be reproduced locally. As for whether we should add the HDFS-7933 fsck output as a configurable option, Akira Ajisaka Vinod Kumar Vavilapalli Konstantin Shvachko any opinions?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 12m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 38s branch-2.7 passed
        +1 compile 1m 7s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 6s branch-2.7 passed with JDK v1.7.0_131
        +1 checkstyle 0m 26s branch-2.7 passed
        +1 mvnsite 1m 4s branch-2.7 passed
        +1 findbugs 3m 9s branch-2.7 passed
        +1 javadoc 1m 1s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 52s branch-2.7 passed with JDK v1.7.0_131
        +1 mvninstall 1m 1s the patch passed
        +1 compile 1m 2s the patch passed with JDK v1.8.0_131
        +1 javac 1m 2s the patch passed
        +1 compile 1m 4s the patch passed with JDK v1.7.0_131
        +1 javac 1m 4s the patch passed
        -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85)
        +1 mvnsite 1m 0s the patch passed
        -1 whitespace 0m 0s The patch has 1304 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 32s The patch 70 line(s) with tabs.
        +1 findbugs 3m 19s the patch passed
        +1 javadoc 1m 1s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 48s the patch passed with JDK v1.7.0_131
        -1 unit 45m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        135m 52s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.TestReplaceDatanodeOnFailure
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.TestPersistBlocks
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.datanode.TestBlockReplacement



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:67e87c9
        JIRA Issue HDFS-11743
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871480/HDFS-11743-branch-2.7.00.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 08514e0aebc7 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / db81fbb
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19785/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19785/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 38s branch-2.7 passed +1 compile 1m 7s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 6s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 26s branch-2.7 passed +1 mvnsite 1m 4s branch-2.7 passed +1 findbugs 3m 9s branch-2.7 passed +1 javadoc 1m 1s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 52s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 1m 1s the patch passed +1 compile 1m 2s the patch passed with JDK v1.8.0_131 +1 javac 1m 2s the patch passed +1 compile 1m 4s the patch passed with JDK v1.7.0_131 +1 javac 1m 4s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 4 fixed = 82 total (was 85) +1 mvnsite 1m 0s the patch passed -1 whitespace 0m 0s The patch has 1304 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 32s The patch 70 line(s) with tabs. +1 findbugs 3m 19s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_131 -1 unit 45m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 135m 52s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestReplaceDatanodeOnFailure JDK v1.7.0_131 Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11743 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871480/HDFS-11743-branch-2.7.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 08514e0aebc7 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / db81fbb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19785/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19785/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19785/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Zhe Zhang thanks for reporting and working on this.

        Should we return only decommissioned replicas(like before HDFS-7933)..? May be you can just return decommissioned in NumberReplicas#decommissionedReplicas..?

        248	      out.println("No. of decommission Replica: "
        249	          + numberReplicas.decommissionedAndDecommissioning());
        

        And as I mentioned in HDFS-7933, we can make it configurable in seperate jira..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - Zhe Zhang thanks for reporting and working on this. Should we return only decommissioned replicas(like before HDFS-7933 )..? May be you can just return decommissioned in NumberReplicas#decommissionedReplicas ..? 248 out.println( "No. of decommission Replica: " 249 + numberReplicas.decommissionedAndDecommissioning()); And as I mentioned in HDFS-7933 , we can make it configurable in seperate jira..?
        Hide
        zhz Zhe Zhang added a comment - - edited

        Attaching v00 patch to revert NamenodeFsck and TestFsck.

        Show
        zhz Zhe Zhang added a comment - - edited Attaching v00 patch to revert NamenodeFsck and TestFsck .
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Sounds good. Thanks Zhe Zhang.

        Show
        ajisakaa Akira Ajisaka added a comment - Sounds good. Thanks Zhe Zhang .
        Hide
        zhz Zhe Zhang added a comment -

        Per latest discussion under HDFS-7933, repurposing this JIRA to revert the incompatible fsck output (but keep the calculation logic behind decommissioning and decommissioned replicas).

        Akira Ajisaka, Vinod Kumar Vavilapalli LMK if the above plan sounds good.

        Show
        zhz Zhe Zhang added a comment - Per latest discussion under HDFS-7933 , repurposing this JIRA to revert the incompatible fsck output (but keep the calculation logic behind decommissioning and decommissioned replicas). Akira Ajisaka , Vinod Kumar Vavilapalli LMK if the above plan sounds good.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1 for reverting HDFS-7933 since it has changed the output format of hdfs fsck command.

        Show
        ajisakaa Akira Ajisaka added a comment - +1 for reverting HDFS-7933 since it has changed the output format of hdfs fsck command.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Zhe Zhang, I'm pushing for a 2.7.4, can you explain why it has to be reverted?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Zhe Zhang , I'm pushing for a 2.7.4, can you explain why it has to be reverted?

          People

          • Assignee:
            zhz Zhe Zhang
            Reporter:
            zhz Zhe Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development