Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Introduced new dfsadmin command saveNamespace to command the name service to do an immediate save of the file system image.

      Description

      It would be useful to have an admin command that saves current namespace.
      This command can be used before regular (planned) cluster shutdown.
      The command will save the namespace into storage directory(s) and reset the name-node journal (edits file).
      It will also reduce name-node startup time, because edits do not need to be digest.

      1. saveNamespace.patch
        14 kB
        Konstantin Shvachko
      2. saveNamespace.patch
        14 kB
        Konstantin Shvachko
      3. saveNamespace.patch
        15 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          saveNamespace will save the namespace image directly to disk(s), it does not need to replay the journal.
          Since saving the image is much faster than digesting the edits the command can substantially reduce the overall cluster restart time.

          Recommended procedure for restarting the cluster:

          1. enter safe mode
          2. save namespace
          3. shutdown the cluster
          4. start the cluster

          The patch introduces a new DFSAdmin command which is called using

          hadoop dfsadmin -saveNamespace
          

          As all other DFSAdmin commands it requires superuser permissions.
          In addition, the name-node must be in safe mode, because we don't want to allow changing namespace during the save.
          In order to enter safe mode call

          hadoop dfsadmin -safemode enter
          

          The patch also corrects 2 warnings in TestCheckpoint, and 2 Javadoc warnings in FSNamesystem.

          Show
          Konstantin Shvachko added a comment - saveNamespace will save the namespace image directly to disk(s), it does not need to replay the journal. Since saving the image is much faster than digesting the edits the command can substantially reduce the overall cluster restart time. Recommended procedure for restarting the cluster: enter safe mode save namespace shutdown the cluster start the cluster The patch introduces a new DFSAdmin command which is called using hadoop dfsadmin -saveNamespace As all other DFSAdmin commands it requires superuser permissions . In addition, the name-node must be in safe mode , because we don't want to allow changing namespace during the save. In order to enter safe mode call hadoop dfsadmin -safemode enter The patch also corrects 2 warnings in TestCheckpoint, and 2 Javadoc warnings in FSNamesystem.
          Hide
          Suresh Srinivas added a comment -
          • In DistributedFileSystem.java, not sure why DFSClient is made a public member. The code compiles without it.
          • In DFSAdmin.refreshNodes, deprecated method fs.getName() can be replaced with fs.getUri();
          Show
          Suresh Srinivas added a comment - In DistributedFileSystem.java, not sure why DFSClient is made a public member. The code compiles without it. In DFSAdmin.refreshNodes, deprecated method fs.getName() can be replaced with fs.getUri();
          Hide
          Suresh Srinivas added a comment -

          Should we have a version of stopping the namenode gracefully, which automatically does the manual steps proposed in this change?

          Show
          Suresh Srinivas added a comment - Should we have a version of stopping the namenode gracefully, which automatically does the manual steps proposed in this change?
          Hide
          Konstantin Shvachko added a comment -

          Forgot to remove public after doing some testing. Change getName() to getURI(). This makes sense since I fixed 2 warnings anyway.
          I was also thinking about a real shutdown command, don't think we should introduce it now.

          Show
          Konstantin Shvachko added a comment - Forgot to remove public after doing some testing. Change getName() to getURI(). This makes sense since I fixed 2 warnings anyway. I was also thinking about a real shutdown command, don't think we should introduce it now.
          Hide
          Lohit Vijayarenu added a comment -

          This looks good. One minor comment if it would help. In the testcase, it would be good to create files, save namespace, stop namenode, start namenode and validate files.

          Show
          Lohit Vijayarenu added a comment - This looks good. One minor comment if it would help. In the testcase, it would be good to create files, save namespace, stop namenode, start namenode and validate files.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • Patch cannot be applied to trunk.
          • DFSClient.saveNamespace() should not be public.
          • Update ClientProtocol version
          • In the javadoc of FSNamesystem.saveNamespace(),
            • Change "superuser permission" to "superuser privilege"
            • Change "permission status is not superuser" to "current user is not a superuser"
          • Should update forrest doc.
          Show
          Tsz Wo Nicholas Sze added a comment - Patch cannot be applied to trunk. DFSClient.saveNamespace() should not be public. Update ClientProtocol version In the javadoc of FSNamesystem.saveNamespace(), Change "superuser permission" to "superuser privilege" Change "permission status is not superuser" to "current user is not a superuser" Should update forrest doc.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > In addition, the name-node must be in safe mode, because we don't want to allow changing namespace during the save.

          Safe mode may not be required since the namespace lock is acquired before saving images. Consider that saving images may take a long time, it makes sense to set safe mode.

          Show
          Tsz Wo Nicholas Sze added a comment - > In addition, the name-node must be in safe mode, because we don't want to allow changing namespace during the save. Safe mode may not be required since the namespace lock is acquired before saving images. Consider that saving images may take a long time, it makes sense to set safe mode.
          Hide
          Konstantin Shvachko added a comment - - edited
          1. Merged with current trunk.
          2. Incorporated Lohit's comments.
          3. In addition made the test to verify that the edits file is not empty after adding a new file, but is empty after saving the namespace.
          4. incorporated Nicholas's comments.
          5. Also included unwrapping of the AccessControlException in DFSClient.
          6. I'd prefer to change documentation in a follow up issue.
          Show
          Konstantin Shvachko added a comment - - edited Merged with current trunk. Incorporated Lohit's comments. In addition made the test to verify that the edits file is not empty after adding a new file, but is empty after saving the namespace. incorporated Nicholas's comments. Also included unwrapping of the AccessControlException in DFSClient. I'd prefer to change documentation in a follow up issue.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good. Below are the ant test-patch results.

               [exec] +1 overall.  
          
               [exec]     +1 @author.  The patch does not contain any @author tags.
          
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
          
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
          
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          
          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. Below are the ant test-patch results. [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.
          In general we do not promote new features to earlier releases.
          But if there is a need to do that with this one let me know.

          Show
          Konstantin Shvachko added a comment - I just committed this. In general we do not promote new features to earlier releases. But if there is a need to do that with this one let me know.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > In general we do not promote new features to earlier releases.
          > But if there is a need to do that with this one let me know.

          Even there is a need, we could not promote this to earlier releases since this is a incompatible change.

          Show
          Tsz Wo Nicholas Sze added a comment - > In general we do not promote new features to earlier releases. > But if there is a need to do that with this one let me know. Even there is a need, we could not promote this to earlier releases since this is a incompatible change.
          Hide
          Robert Chansler added a comment -

          Edit release note for publication.

          Show
          Robert Chansler added a comment - Edit release note for publication.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > 6. I'd prefer to change documentation in a follow up issue.

          Hi Konstantin, please create a follow up issue if you haven't.

          Show
          Tsz Wo Nicholas Sze added a comment - > 6. I'd prefer to change documentation in a follow up issue. Hi Konstantin, please create a follow up issue if you haven't.

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development