Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2585

Automatic namespace recovery from the secondary image.

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.16.0
    • 0.18.0
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Improved management of replicas of the name space image. If all replicas on the Name Node are lost, the latest check point can be loaded from the secondary Name Node. Use parameter "-importCheckpoint" and specify the location with "fs.checkpoint.dir." The directory structure on the secondary Name Node has changed to match the primary Name Node.
      Show
      Improved management of replicas of the name space image. If all replicas on the Name Node are lost, the latest check point can be loaded from the secondary Name Node. Use parameter "-importCheckpoint" and specify the location with "fs.checkpoint.dir." The directory structure on the secondary Name Node has changed to match the primary Name Node.

    Description

      Hadoop has a three way (configuration controlled) protection from loosing the namespace image.

      1. image can be replicated on different hard-drives of the same node;
      2. image can be replicated on a nfs mounted drive on an independent node;
      3. a stale replica of the image is created during periodic checkpointing and stored on the secondary name-node.

      Currently during startup the name-node examines all configured storage directories, selects the
      most up to date image, reads it, merges with the corresponding edits, and writes to the new image back
      into all storage directories. Everything is done automatically.

      If due to multiple hardware failures none of those images on mounted hard drives (local or remote)
      are available the secondary image although stale (up to one hour old by default) can be still
      used in order to recover the majority of the file system data.
      Currently one can reconstruct a valid name-node image from the secondary one manually.
      It would be nice to support an automatic recovery.

      Attachments

        1. SecondaryStorage.patch
          68 kB
          Konstantin Shvachko
        2. SecondaryStorage.patch
          68 kB
          Konstantin Shvachko
        3. SecondaryStorage.patch
          70 kB
          Konstantin Shvachko

        Issue Links

          Activity

            People

              shv Konstantin Shvachko
              shv Konstantin Shvachko
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: