Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1509

Resync discarded directories in fs.name.dir during saveNamespace command

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.

      1. resyncBadNameDir3.txt
        8 kB
        dhruba borthakur
      2. resyncBadNameDir2.txt
        7 kB
        dhruba borthakur
      3. resyncBadNameDir1.txt
        9 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          If a fs.name.dir is faulty (eg a failed local disk or flaky NFS mount) then won't this mean we continually fail writing fsedits, unless you dynamically update the configuration to remove the failed fs.name.dir?

          The idea behind HADOOP-4885 was that a faulty dir gets black-listed will get re-instated on the first checkpoint after it becomes valid. Is this approach insufficient?

          Show
          Eli Collins added a comment - If a fs.name.dir is faulty (eg a failed local disk or flaky NFS mount) then won't this mean we continually fail writing fsedits, unless you dynamically update the configuration to remove the failed fs.name.dir? The idea behind HADOOP-4885 was that a faulty dir gets black-listed will get re-instated on the first checkpoint after it becomes valid. Is this approach insufficient?
          Hide
          dhruba borthakur added a comment -

          > dir gets black-listed will get re-instated on the first checkpoint after it becomes valid.

          This process works ok. This is just another manual way of triggering this sooner (rather than waiting for the next checkpoint)

          Show
          dhruba borthakur added a comment - > dir gets black-listed will get re-instated on the first checkpoint after it becomes valid. This process works ok. This is just another manual way of triggering this sooner (rather than waiting for the next checkpoint)
          Hide
          Eli Collins added a comment -

          Isn't there a motivation for the Namenode no longer writing new edits to a directory when it encounters an error while writing to a fs.name.dir directory? eg that constantly try writing to a failed directory slows things down.

          Show
          Eli Collins added a comment - Isn't there a motivation for the Namenode no longer writing new edits to a directory when it encounters an error while writing to a fs.name.dir directory? eg that constantly try writing to a failed directory slows things down.
          Hide
          dhruba borthakur added a comment -

          Absolutely agree that constantly trying to write to a failed directory will slow things down, I am not suggesting we do this. Instead, "bin/hadoop dfsadmin -savenamespace" is a command line utility and is likely to be run manually by an administrator; when this command is run the namenode saves its entire image from memory in fsimage (and truncates fsedits). I would like this operation to try writing the fsimage to all configured directories.

          Show
          dhruba borthakur added a comment - Absolutely agree that constantly trying to write to a failed directory will slow things down, I am not suggesting we do this. Instead, "bin/hadoop dfsadmin -savenamespace" is a command line utility and is likely to be run manually by an administrator; when this command is run the namenode saves its entire image from memory in fsimage (and truncates fsedits). I would like this operation to try writing the fsimage to all configured directories.
          Hide
          Eli Collins added a comment -

          Ah, thanks. In the description I think you mean fsimage rather than edits then.

          Show
          Eli Collins added a comment - Ah, thanks. In the description I think you mean fsimage rather than edits then.
          Hide
          dhruba borthakur added a comment -

          Unit test doe snot pass yet, this is a draft version of the code.

          Show
          dhruba borthakur added a comment - Unit test doe snot pass yet, this is a draft version of the code.
          Hide
          dhruba borthakur added a comment -

          With a little help from Hairong, I could write a unit test using the mockito framework to test this out. The actual code fix is only one line because the hard-work of resync bad edits was already done by Boris via HADOOP-4885.

          Review comments welcome.

          Show
          dhruba borthakur added a comment - With a little help from Hairong, I could write a unit test using the mockito framework to test this out. The actual code fix is only one line because the hard-work of resync bad edits was already done by Boris via HADOOP-4885 . Review comments welcome.
          Hide
          Eli Collins added a comment -

          +1 lgtm. nice test.

          Show
          Eli Collins added a comment - +1 lgtm. nice test.
          Hide
          Hairong Kuang added a comment -

          I agree with Eli that this looks good except for a couple of improvements:
          1. Could restore the accidentally commented-out tests?
          2. attemptRestoreRemovedStorage formats the newly added previously removed storage directories, which calls saveCurrent that save fsimage to disk. Later on saveNamespace save the namespace again. Could you make name space saving to be done only once?

          Show
          Hairong Kuang added a comment - I agree with Eli that this looks good except for a couple of improvements: 1. Could restore the accidentally commented-out tests? 2. attemptRestoreRemovedStorage formats the newly added previously removed storage directories, which calls saveCurrent that save fsimage to disk. Later on saveNamespace save the namespace again. Could you make name space saving to be done only once?
          Hide
          dhruba borthakur added a comment -

          I incorporated both of Hairong's comments. Thanks Hairong and Eli for reviewing this patch.

          Show
          dhruba borthakur added a comment - I incorporated both of Hairong's comments. Thanks Hairong and Eli for reviewing this patch.
          Hide
          Konstantin Shvachko added a comment -

          I did not understand the description for this jira. Eli is mentioning you use edits instead of image or vice versa. Could you please edit the description to clarify what is proposed.

          Show
          Konstantin Shvachko added a comment - I did not understand the description for this jira. Eli is mentioning you use edits instead of image or vice versa. Could you please edit the description to clarify what is proposed.
          Hide
          Hairong Kuang added a comment -

          +1. The patch looks good to me.

          Show
          Hairong Kuang added a comment - +1. The patch looks good to me.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12466375/resyncBadNameDir3.txt
          against trunk revision 1051669.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestHDFSTrash

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12466375/resyncBadNameDir3.txt against trunk revision 1051669. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestHDFSTrash -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          The four failed tests are the usual ones:

          org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransferVerySmallWrite
          org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransfer
          org.apache.hadoop.hdfs.TestHDFSTrash.testTrashEmptier
          org.apache.hadoop.hdfs.server.namenode.TestStorageRestore.testStorageRestore

          Show
          dhruba borthakur added a comment - The four failed tests are the usual ones: org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransferVerySmallWrite org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransfer org.apache.hadoop.hdfs.TestHDFSTrash.testTrashEmptier org.apache.hadoop.hdfs.server.namenode.TestStorageRestore.testStorageRestore
          Hide
          dhruba borthakur added a comment -

          @Konstanin: This patch makes the savenamespace command write the fsimage and edits into all configured directories.

          Show
          dhruba borthakur added a comment - @Konstanin: This patch makes the savenamespace command write the fsimage and edits into all configured directories.
          Hide
          dhruba borthakur added a comment -

          I just committed this.

          Show
          dhruba borthakur added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development