Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11009

Add a tool to reconstruct block meta file from CLI

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If the block file on local disk presents, but the meta file is missing, it's theoretically possible to manually restore the meta file and have the block readable.
      This jira is to propose to add such a tool to do that.

      1. HDFS-11090.branch-2.patch
        23 kB
        Xiao Chen
      2. HDFS-11009.04.patch
        18 kB
        Xiao Chen
      3. HDFS-11009.03.patch
        17 kB
        Xiao Chen
      4. HDFS-11009.02.patch
        14 kB
        Xiao Chen
      5. HDFS-11009.01.patch
        12 kB
        Xiao Chen

        Activity

        Hide
        xiaochen Xiao Chen added a comment -

        Patch 1 to add the feature to debug admin. Manually tested with a block file using default crc, tool output matches with the block meta file.

        Show
        xiaochen Xiao Chen added a comment - Patch 1 to add the feature to debug admin. Manually tested with a block file using default crc, tool output matches with the block meta file.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 30s Maven dependency ordering for branch
        +1 mvninstall 7m 49s trunk passed
        +1 compile 1m 26s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 1m 32s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 26s trunk passed
        +1 javadoc 1m 0s trunk passed
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 1m 29s the patch passed
        +1 javac 1m 29s the patch passed
        -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 4 new + 97 unchanged - 0 fixed = 101 total (was 97)
        +1 mvnsite 1m 38s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 24s the patch passed
        +1 javadoc 0m 56s the patch passed
        +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
        -1 unit 70m 57s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        100m 22s



        Reason Tests
        Failed junit tests hadoop.hdfs.tools.TestDebugAdmin



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11009
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833325/HDFS-11009.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3fe1bb4c3ca4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dbe663d
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17154/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17154/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17154/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17154/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 7m 49s trunk passed +1 compile 1m 26s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 26s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 1m 29s the patch passed +1 javac 1m 29s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 4 new + 97 unchanged - 0 fixed = 101 total (was 97) +1 mvnsite 1m 38s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 70m 57s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 100m 22s Reason Tests Failed junit tests hadoop.hdfs.tools.TestDebugAdmin Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11009 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833325/HDFS-11009.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3fe1bb4c3ca4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbe663d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17154/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17154/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17154/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17154/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 2 ready for review:

        • Fixed checkstyle and test failure
        • Added documentation
        Show
        xiaochen Xiao Chen added a comment - Patch 2 ready for review: Fixed checkstyle and test failure Added documentation
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 6s Maven dependency ordering for branch
        +1 mvninstall 6m 55s trunk passed
        +1 compile 1m 24s trunk passed
        +1 checkstyle 0m 32s trunk passed
        +1 mvnsite 1m 26s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 17s trunk passed
        +1 javadoc 1m 2s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 1m 31s the patch passed
        +1 javac 1m 31s the patch passed
        +1 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 0 new + 94 unchanged - 3 fixed = 94 total (was 97)
        +1 mvnsite 1m 29s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 29s the patch passed
        +1 javadoc 0m 57s the patch passed
        +1 unit 0m 59s hadoop-hdfs-client in the patch passed.
        -1 unit 84m 28s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        112m 29s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11009
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833420/HDFS-11009.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e9845a232305 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 701c27a
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17162/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17162/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17162/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 6m 55s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 17s trunk passed +1 javadoc 1m 2s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 1m 31s the patch passed +1 javac 1m 31s the patch passed +1 checkstyle 0m 29s hadoop-hdfs-project: The patch generated 0 new + 94 unchanged - 3 fixed = 94 total (was 97) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 29s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 0m 59s hadoop-hdfs-client in the patch passed. -1 unit 84m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 112m 29s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11009 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833420/HDFS-11009.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e9845a232305 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 701c27a Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17162/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17162/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17162/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for working on this Xiao. Looks good overall, though a few review comments:

        • Looks like both the block and out files are required parameters, so no please do not use "[]" in the help text and doc. It's also debatable whether these should be flags at all, since UNIX normally uses positional args if they're required.
        • I'm mixed on this one, but it'd be more clear if this command was named "computeMeta". Unfortunately there's also the existing "verify" command, and the commands are named "*BlockChecksumCommand". The debug command comes with no compat guarantees, so maybe we do some renames even for branch-2.
        • It'd also be nice to add some disclaimers to the doc/help text. I'm guessing this is meant for use in emergency situations where the block only has corrupt replicas? I'm also wondering about the danger of divergent state on different DNs, since I don't think HDFS-level mechanisms like the BlockScanner or FBRs will catch this.
        Show
        andrew.wang Andrew Wang added a comment - Thanks for working on this Xiao. Looks good overall, though a few review comments: Looks like both the block and out files are required parameters, so no please do not use "[]" in the help text and doc. It's also debatable whether these should be flags at all, since UNIX normally uses positional args if they're required. I'm mixed on this one, but it'd be more clear if this command was named "computeMeta". Unfortunately there's also the existing "verify" command, and the commands are named "*BlockChecksumCommand". The debug command comes with no compat guarantees, so maybe we do some renames even for branch-2. It'd also be nice to add some disclaimers to the doc/help text. I'm guessing this is meant for use in emergency situations where the block only has corrupt replicas? I'm also wondering about the danger of divergent state on different DNs, since I don't think HDFS-level mechanisms like the BlockScanner or FBRs will catch this.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew for reviewing! Patch 3 to address most comments, with some debate/explanation:

        It's also debatable whether these should be flags at all, since UNIX normally uses positional args if they're required.

        Good point. I'm keeping it for consistency with other debug commands: -meta for verifyMeta and -path for recoverLease are flagged. Please let me know if you feel strongly.

        I'm guessing this is meant for use in emergency situations where the block only has corrupt replicas? I'm also wondering about the danger of divergent state on different DNs, since I don't think HDFS-level mechanisms like the BlockScanner or FBRs will catch this.

        Exactly. So if there's no good replicas, but the HDFS file is critical, and the blockfile is not deleted on disk, there's a chance to restore the .meta file. One still need to search in the logs carefully to find genStamp, but it's theoretically doable.
        I don't think BlockScanner would catch this and work, but I tried restarting DN locally (in a single DN env), and after FBR the block is readable again. It's not a 'divergent state' in this case, just DNs with corrupt replicas and good replicas, right?

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew for reviewing! Patch 3 to address most comments, with some debate/explanation: It's also debatable whether these should be flags at all, since UNIX normally uses positional args if they're required. Good point. I'm keeping it for consistency with other debug commands: -meta for verifyMeta and -path for recoverLease are flagged. Please let me know if you feel strongly. I'm guessing this is meant for use in emergency situations where the block only has corrupt replicas? I'm also wondering about the danger of divergent state on different DNs, since I don't think HDFS-level mechanisms like the BlockScanner or FBRs will catch this. Exactly. So if there's no good replicas, but the HDFS file is critical, and the blockfile is not deleted on disk, there's a chance to restore the .meta file. One still need to search in the logs carefully to find genStamp, but it's theoretically doable. I don't think BlockScanner would catch this and work, but I tried restarting DN locally (in a single DN env), and after FBR the block is readable again. It's not a 'divergent state' in this case, just DNs with corrupt replicas and good replicas, right?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 8s Maven dependency ordering for branch
        +1 mvninstall 7m 12s trunk passed
        +1 compile 1m 27s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 1m 27s trunk passed
        +1 mvneclipse 0m 24s trunk passed
        +1 findbugs 3m 6s trunk passed
        +1 javadoc 0m 59s trunk passed
        0 mvndep 0m 6s Maven dependency ordering for patch
        +1 mvninstall 1m 12s the patch passed
        +1 compile 1m 16s the patch passed
        +1 javac 1m 16s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project: The patch generated 1 new + 94 unchanged - 3 fixed = 95 total (was 97)
        +1 mvnsite 1m 18s the patch passed
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 12s the patch passed
        +1 javadoc 0m 55s the patch passed
        +1 unit 0m 53s hadoop-hdfs-client in the patch passed.
        -1 unit 58m 12s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        84m 56s



        Reason Tests
        Failed junit tests hadoop.tracing.TestTracing



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11009
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833891/HDFS-11009.03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 55f4dffad050 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c023c74
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17198/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17198/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17198/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17198/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 7m 12s trunk passed +1 compile 1m 27s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 0m 59s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 1m 16s the patch passed +1 javac 1m 16s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project: The patch generated 1 new + 94 unchanged - 3 fixed = 95 total (was 97) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 0m 53s hadoop-hdfs-client in the patch passed. -1 unit 58m 12s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 84m 56s Reason Tests Failed junit tests hadoop.tracing.TestTracing Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11009 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833891/HDFS-11009.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 55f4dffad050 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c023c74 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17198/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17198/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17198/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17198/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the rev Xiao,

        Good point. I'm keeping it for consistency with other debug commands: -meta for verifyMeta and -path for recoverLease are flagged. Please let me know if you feel strongly.

        I don't feel strongly, and honestly this pattern is prevalent enough throughout other HDFS commands that I think it's better to maintain for consistency.

        I don't think BlockScanner would catch this and work, but I tried restarting DN locally (in a single DN env), and after FBR the block is readable again. It's not a 'divergent state' in this case, just DNs with corrupt replicas and good replicas, right?

        My concern was that if we regenerate the meta file for a corrupted block, it won't show up as corrupted. AFAIK the DN does not compare its checksums against other DNs, so we would have a replica that shows as good, but is in fact corrupted. If I'm right here, it'd be nice to discuss this a bit in the doc/help.

        Few other nits:

        • Could we fix the verifyMeta help text while we're here to also remove the "[]"? The "-block" flag looks required to me.
        • "If a block file is specified, we will compute the checksums from the block file, and save it to the specified output metadata file." Since it says "If", it makes it sound like specifying a block file is optional, but it's actually required.
        Show
        andrew.wang Andrew Wang added a comment - Thanks for the rev Xiao, Good point. I'm keeping it for consistency with other debug commands: -meta for verifyMeta and -path for recoverLease are flagged. Please let me know if you feel strongly. I don't feel strongly, and honestly this pattern is prevalent enough throughout other HDFS commands that I think it's better to maintain for consistency. I don't think BlockScanner would catch this and work, but I tried restarting DN locally (in a single DN env), and after FBR the block is readable again. It's not a 'divergent state' in this case, just DNs with corrupt replicas and good replicas, right? My concern was that if we regenerate the meta file for a corrupted block, it won't show up as corrupted. AFAIK the DN does not compare its checksums against other DNs, so we would have a replica that shows as good, but is in fact corrupted. If I'm right here, it'd be nice to discuss this a bit in the doc/help. Few other nits: Could we fix the verifyMeta help text while we're here to also remove the "[]"? The "-block" flag looks required to me. "If a block file is specified, we will compute the checksums from the block file, and save it to the specified output metadata file." Since it says "If", it makes it sound like specifying a block file is optional, but it's actually required.
        Hide
        xiaochen Xiao Chen added a comment - - edited

        Thanks Andrew for explaining and more reviews.

        I see your point now... added more text to make this scarier and more accurate. How does it feel now? The 'if' text is also fixed

        CLI output looks like this now:

        xiao$ bin/hdfs debug computeMeta
        computeMeta -block <block-file> -out <output-metadata-file>
          Compute HDFS metadata from the specified block file, and save it to
          the specified output metadata file.
        
        **NOTE: Use at your own risk!
         If the block file is corrupt and you overwrite it's meta file, 
         it will show up as good in HDFS, but you can't read the data.
         Only use as a last measure, and when you are 100% certain the block file is good.
        

        The assumption here is that when people is running the debug command, they know what they're doing. (At least need to figure out which block file from which HDFS file is of interest, and where that block file locates etc.)

        Could we fix the verifyMeta help text while we're here to also remove the "[]"? The "-block" flag looks required to me.

        Not really, so if you verify a meta file, it checks it can be read as a meta. If you also give a block file, it then verifies checksum matches.

        Show
        xiaochen Xiao Chen added a comment - - edited Thanks Andrew for explaining and more reviews. I see your point now... added more text to make this scarier and more accurate. How does it feel now? The 'if' text is also fixed CLI output looks like this now: xiao$ bin/hdfs debug computeMeta computeMeta -block <block-file> -out <output-metadata-file> Compute HDFS metadata from the specified block file, and save it to the specified output metadata file. **NOTE: Use at your own risk! If the block file is corrupt and you overwrite it's meta file, it will show up as good in HDFS, but you can't read the data. Only use as a last measure, and when you are 100% certain the block file is good. The assumption here is that when people is running the debug command, they know what they're doing. (At least need to figure out which block file from which HDFS file is of interest, and where that block file locates etc.) Could we fix the verifyMeta help text while we're here to also remove the "[]"? The "-block" flag looks required to me. Not really, so if you verify a meta file, it checks it can be read as a meta. If you also give a block file, it then verifies checksum matches.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 7s Maven dependency ordering for branch
        +1 mvninstall 8m 14s trunk passed
        +1 compile 1m 51s trunk passed
        +1 checkstyle 0m 34s trunk passed
        +1 mvnsite 1m 31s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 45s trunk passed
        +1 javadoc 1m 10s trunk passed
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 41s the patch passed
        +1 compile 1m 50s the patch passed
        +1 javac 1m 50s the patch passed
        +1 checkstyle 0m 35s hadoop-hdfs-project: The patch generated 0 new + 94 unchanged - 3 fixed = 94 total (was 97)
        +1 mvnsite 1m 41s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 25s the patch passed
        +1 javadoc 0m 58s the patch passed
        +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
        +1 unit 77m 41s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        109m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11009
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834026/HDFS-11009.04.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux add396af56e8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b733a6f
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17208/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17208/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 8m 14s trunk passed +1 compile 1m 51s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 45s trunk passed +1 javadoc 1m 10s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 41s the patch passed +1 compile 1m 50s the patch passed +1 javac 1m 50s the patch passed +1 checkstyle 0m 35s hadoop-hdfs-project: The patch generated 0 new + 94 unchanged - 3 fixed = 94 total (was 97) +1 mvnsite 1m 41s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 25s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 0m 57s hadoop-hdfs-client in the patch passed. +1 unit 77m 41s hadoop-hdfs in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 109m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11009 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834026/HDFS-11009.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux add396af56e8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b733a6f Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17208/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17208/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM thanks Xiao!

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM thanks Xiao!
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew!
        Backporting to branch-2 has some minor conflict due to FSDatasetImpl change from HDFS-10636. Uploading a branch-2 patch.

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew! Backporting to branch-2 has some minor conflict due to FSDatasetImpl change from HDFS-10636 . Uploading a branch-2 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 30s Maven dependency ordering for branch
        +1 mvninstall 6m 45s branch-2 passed
        +1 compile 1m 19s branch-2 passed with JDK v1.8.0_101
        +1 compile 1m 22s branch-2 passed with JDK v1.7.0_111
        +1 checkstyle 0m 35s branch-2 passed
        +1 mvnsite 1m 25s branch-2 passed
        +1 mvneclipse 0m 29s branch-2 passed
        +1 findbugs 3m 34s branch-2 passed
        +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_101
        +1 javadoc 2m 0s branch-2 passed with JDK v1.7.0_111
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 14s the patch passed
        +1 compile 1m 11s the patch passed with JDK v1.8.0_101
        +1 javac 1m 11s the patch passed
        +1 compile 1m 22s the patch passed with JDK v1.7.0_111
        +1 javac 1m 22s the patch passed
        +1 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 0 new + 128 unchanged - 3 fixed = 128 total (was 131)
        +1 mvnsite 1m 20s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 56s the patch passed
        +1 javadoc 1m 8s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 54s the patch passed with JDK v1.7.0_111
        +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_111.
        -1 unit 52m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
        +1 asflicense 0m 28s The patch does not generate ASF License warnings.
        143m 43s



        Reason Tests
        JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
        JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestFileCorruption
          hadoop.hdfs.TestRenameWhileOpen



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue HDFS-11009
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834083/HDFS-11090.branch-2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fb4efdf28968 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 74f1c49
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17211/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
        JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17211/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17211/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 6m 45s branch-2 passed +1 compile 1m 19s branch-2 passed with JDK v1.8.0_101 +1 compile 1m 22s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 35s branch-2 passed +1 mvnsite 1m 25s branch-2 passed +1 mvneclipse 0m 29s branch-2 passed +1 findbugs 3m 34s branch-2 passed +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_101 +1 javadoc 2m 0s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 1m 11s the patch passed with JDK v1.8.0_101 +1 javac 1m 11s the patch passed +1 compile 1m 22s the patch passed with JDK v1.7.0_111 +1 javac 1m 22s the patch passed +1 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 0 new + 128 unchanged - 3 fixed = 128 total (was 131) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 56s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 54s the patch passed with JDK v1.7.0_111 +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_111. -1 unit 52m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 143m 43s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestFileCorruption   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11009 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834083/HDFS-11090.branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fb4efdf28968 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 74f1c49 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17211/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17211/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17211/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Test failures are unrelated.
        Committed this to trunk, branch-2 and branch-2.8. Thanks Andrew for the reviews!

        Show
        xiaochen Xiao Chen added a comment - Test failures are unrelated. Committed this to trunk, branch-2 and branch-2.8. Thanks Andrew for the reviews!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10637 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10637/)
        HDFS-11009. Add a tool to reconstruct block meta file from CLI. (xiao: rev c5573e6a7599da17cad733cd274e7a9b75b22bb0)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDebugAdmin.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10637 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10637/ ) HDFS-11009 . Add a tool to reconstruct block meta file from CLI. (xiao: rev c5573e6a7599da17cad733cd274e7a9b75b22bb0) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDebugAdmin.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java

          People

          • Assignee:
            xiaochen Xiao Chen
            Reporter:
            xiaochen Xiao Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development