Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: namenode
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fsck now reports whether a file is replicated and erasure-coded. If it is replicated, fsck reports replication factor of the file. If it is erasure coded, fsck reports the erasure coding policy of the file.

      Description

      Currently fsck reports corrupt EC files as follows: it does not distinguish erasure coded files versus replicated files. In addition, it would be nice to print out the ec policy of the corrupt EC file.

      /striped <dir>
      /striped/corrupted 393216 bytes, 1 block(s):
      /striped/corrupted: CORRUPT blockpool BP-1564681138-127.0.0.1-1475793860787 block blk_-9223372036854775792
      Under replicated BP-1564681138-127.0.0.1-1475793860787:blk_-9223372036854775792_1001. Target Replicas is 9 but found 5 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
      CORRUPT 1 blocks of total size 393216 B
      0. BP-1564681138-127.0.0.1-1475793860787:blk_-9223372036854775792_1001 len=393216 Live_repl=5 [DatanodeInfoWithStorage[127.0.0.1:62192,DS-81dcbc38-755e-446e-a028-71a79e4de6d9,DISK](CORRUPT), DatanodeInfoWithStorage[127.0.0.1:62180,DS-98fe193d-6342-4b2c-ad61-4586e2530b1e,DISK](CORRUPT), DatanodeInfoWithStorage[127.0.0.1:62167,DS-53031f88-0c63-4839-ab18-efea2f1bb063,DISK](CORRUPT), DatanodeInfoWithStorage[127.0.0.1:62162,DS-e8b418fd-165d-4d6f-886b-f75c21be096d,DISK](CORRUPT), DatanodeInfoWithStorage[127.0.0.1:62176,DS-03e51584-5b33-4bb6-89b5-f519cda57429,DISK](LIVE), DatanodeInfoWithStorage[127.0.0.1:62158,DS-ce9ca7b3-5b00-4351-8537-822eed532b46,DISK](LIVE), DatanodeInfoWithStorage[127.0.0.1:62184,DS-668c500e-eb3d-4d40-b900-814076d5e160,DISK](LIVE), DatanodeInfoWithStorage[127.0.0.1:62171,DS-763a3961-b214-4601-81c0-abdaecf539c4,DISK](LIVE), DatanodeInfoWithStorage[127.0.0.1:62188,DS-d4ea6399-bead-452e-8dd5-bc8c5ebd4f45,DISK](LIVE)]

      1. HDFS-10976.002.patch
        3 kB
        Wei-Chiu Chuang
      2. HDFS-10976.001.patch
        3 kB
        Wei-Chiu Chuang

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10648 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10648/)
        HDFS-10976. Report erasure coding policy of EC files in Fsck. (weichiu: rev 5e83a21cb66c78e89ac5af9a130ab0aee596a9f4)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10648 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10648/ ) HDFS-10976 . Report erasure coding policy of EC files in Fsck. (weichiu: rev 5e83a21cb66c78e89ac5af9a130ab0aee596a9f4) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks to Andrew Wang and Takanobu Asanuma for reviewing the patch. Committed the v002 patch to trunk.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks to Andrew Wang and Takanobu Asanuma for reviewing the patch. Committed the v002 patch to trunk.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        +1 (non-binding). Thanks again.

        Show
        tasanuma0829 Takanobu Asanuma added a comment - +1 (non-binding). Thanks again.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Andrew Wang. The failed tests does not fail in my local tree. I'll commit v02 patch.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Andrew Wang . The failed tests does not fail in my local tree. I'll commit v02 patch.
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 40s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 68m 2s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        87m 24s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
          hadoop.hdfs.TestPersistBlocks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10976
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834222/HDFS-10976.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6f500f1e8f33 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e9c4616
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17221/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17221/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17221/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 40s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 68m 2s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 87m 24s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.TestPersistBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10976 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834222/HDFS-10976.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6f500f1e8f33 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e9c4616 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17221/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17221/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17221/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for the review. Attach a v002 patch to address your comment and fix code style warning.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for the review. Attach a v002 patch to address your comment and fix code style warning.
        Hide
        tasanuma0829 Takanobu Asanuma added a comment -

        Thanks for the patch, Wei-Chiu Chuang! It mostly looks good to me(non-binding). One minor comment is that we can reuse the variable.

        ErasureCodingPolicy ecPolicy = file.getErasureCodingPolicy();
        if (ecPolicy == null) { // a replicated file
        
        Show
        tasanuma0829 Takanobu Asanuma added a comment - Thanks for the patch, Wei-Chiu Chuang ! It mostly looks good to me(non-binding). One minor comment is that we can reuse the variable. ErasureCodingPolicy ecPolicy = file.getErasureCodingPolicy(); if (ecPolicy == null ) { // a replicated file
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 26s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 51s trunk passed
        +1 javadoc 0m 45s trunk passed
        +1 mvninstall 0m 55s the patch passed
        +1 compile 0m 48s the patch passed
        +1 javac 0m 48s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 100 unchanged - 0 fixed = 101 total (was 100)
        +1 mvnsite 0m 59s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 4s the patch passed
        +1 javadoc 0m 43s the patch passed
        -1 unit 67m 6s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        88m 29s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10976
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834018/HDFS-10976.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 852aafaa85a2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b733a6f
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17207/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17207/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17207/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17207/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 26s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 100 unchanged - 0 fixed = 101 total (was 100) +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 43s the patch passed -1 unit 67m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 88m 29s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10976 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834018/HDFS-10976.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 852aafaa85a2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b733a6f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17207/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17207/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17207/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17207/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        The output of fsck with -files parameter becomes as follows:

        FSCK started by weichiu (auth:SIMPLE) from /127.0.0.1 for path / at Tue Oct 18 12:15:15 PDT 2016
        / <dir>
        /replicated <dir>
        /replicated/replfile 1024 bytes, replicated: replication=3, 1 block(s):  MISSING 1 blocks of total size 1024 B
        0. BP-1938052379-172.16.1.88-1476818109173:blk_1073741825_1001 len=1024 MISSING!
        
        /striped <dir>
        /striped/largeFile 786432 bytes, erasure-coded: policy=RS-DEFAULT-6-3-64k, 1 block(s):  MISSING 1 blocks of total size 786432 B
        0. BP-1938052379-172.16.1.88-1476818109173:blk_-9223372036854775792_1002 len=786432 MISSING! Live_repl=0
        
        /striped/smallFile 12 bytes, erasure-coded: policy=RS-DEFAULT-6-3-64k, 1 block(s):  MISSING 1 blocks of total size 12 B
        0. BP-1938052379-172.16.1.88-1476818109173:blk_-9223372036854775776_1003 len=12 MISSING! Live_repl=0
        
        
        Show
        jojochuang Wei-Chiu Chuang added a comment - The output of fsck with -files parameter becomes as follows: FSCK started by weichiu (auth:SIMPLE) from /127.0.0.1 for path / at Tue Oct 18 12:15:15 PDT 2016 / <dir> /replicated <dir> /replicated/replfile 1024 bytes, replicated: replication=3, 1 block(s): MISSING 1 blocks of total size 1024 B 0. BP-1938052379-172.16.1.88-1476818109173:blk_1073741825_1001 len=1024 MISSING! /striped <dir> /striped/largeFile 786432 bytes, erasure-coded: policy=RS-DEFAULT-6-3-64k, 1 block(s): MISSING 1 blocks of total size 786432 B 0. BP-1938052379-172.16.1.88-1476818109173:blk_-9223372036854775792_1002 len=786432 MISSING! Live_repl=0 /striped/smallFile 12 bytes, erasure-coded: policy=RS-DEFAULT-6-3-64k, 1 block(s): MISSING 1 blocks of total size 12 B 0. BP-1938052379-172.16.1.88-1476818109173:blk_-9223372036854775776_1003 len=12 MISSING! Live_repl=0
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Post v001 patch. A simple patch to label the "redundancy policy" of a file. Also, add additional assertion for the fsck output.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Post v001 patch. A simple patch to label the "redundancy policy" of a file. Also, add additional assertion for the fsck output.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Assign this jira to myself. I want to take this opportunity to understand fsck implementation.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Assign this jira to myself. I want to take this opportunity to understand fsck implementation.

          People

          • Assignee:
            jojochuang Wei-Chiu Chuang
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development