Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6185 HDFS operational and debuggability improvements
  3. HDFS-6353

Check and make checkpoint before stopping the NameNode

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha1
    • namenode
    • None
    • Incompatible change, Reviewed
    • Stopping the namenode on secure systems now requires the user be authenticated.

    Description

      One of the failure patterns I have seen is, in some rare circumstances, due to some inconsistency the secondary or standby fails to consume editlog. The only solution when this happens is to save the namespace at the current active namenode. But sometimes when this happens, unsuspecting admin might end up restarting the namenode, requiring more complicated solution to the problem (such as ignore editlog record that cannot be consumed etc.).

      How about adding the following functionality:
      When checkpointer (standby or secondary) fails to consume editlog, based on a configurable flag (on/off) to let the active namenode know about this failure. Active namenode can enters safemode and saves namespace. When in this type of safemode, namenode UI also shows information about checkpoint failure and that it is saving namespace. Once the namespace is saved, namenode can come out of safemode.

      This means service unavailability (even in HA cluster). But it might be worth it to avoid long startup times or need for other manual fixes. Thoughts?

      Attachments

        1. HDFS-6353.000.patch
          38 kB
          Jing Zhao
        2. HDFS-6353.001.patch
          43 kB
          Jing Zhao
        3. HDFS-6353.002.branch-2.patch
          43 kB
          Jing Zhao
        4. HDFS-6353.002.branch-2.patch
          43 kB
          Jing Zhao
        5. HDFS-6353.002.patch
          42 kB
          Jing Zhao

        Issue Links

          Activity

            People

              jingzhao Jing Zhao
              sureshms Suresh Srinivas
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: