Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2362 More Improvements on NameNode Scalability
  3. HDFS-5711

Removing memory limitation of the Namenode by persisting Block - Block location mappings to disk.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • namenode
    • None

    Description

      This jira is to track changes to be made to remove HDFS name-node memory limitation to hold block - block location mappings.

      It is a known fact that the single Name-node architecture of HDFS has scalability limits. The HDFS federation project alleviates this problem by using horizontal scaling. This helps increase the throughput of metadata operation and also the amount of data that can be stored in a Hadoop cluster.
      The Name-node stores all the filesystem metadata in memory (even in the federated architecture), the
      Name-node design can be enhanced by persisting part of the metadata onto secondary storage and retaining
      the popular or recently accessed metadata information in main memory. This design can benefit a HDFS deployment
      which doesn't use federation but needs to store a large number of files or large number of blocks. Lin Xiao from Hortonworks attempted a similar
      project [1] in the Summer of 2013. They used LevelDB to persist the Namespace information (i.e file and directory inode information).

      A patch with this change is yet to be submitted to code base. We also intend to use LevelDB to persist metadata, and plan to
      provide a complete solution, by not just persisting the Namespace information but also the Blocks Map onto secondary storage.

      We did implement the basic prototype which stores the block-block location mapping metadata to the persistent key-value store i.e. levelDB. Prototype also maintains the in-memory cache of the recently used block-block location mappings metadata.

      References:
      [1] Lin Xiao, Hortonworks, Removing Name-node’s memory limitation, HDFS-5389, http://www.slideshare.net/ydn/hadoop-meetup-hug-august-2013-removing-the-namenodes-memory-limitation.

      Attachments

        Issue Links

          Activity

            People

              ajithshetty Ajith S
              rohanpasalkar Rohan Pasalkar
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated: