Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1612

HDFS Design Documentation is outdated


    • Hadoop Flags:


      I was trying to discover details about the Secondary NameNode, and came across the description below in the HDFS design doc.

      The NameNode keeps an image of the entire file system namespace and file Blockmap in memory. This key metadata item is designed to be compact, such that a NameNode with 4 GB of RAM is plenty to support a huge number of files and directories. When the NameNode starts up, it reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage. This process is called a checkpoint. In the current implementation, a checkpoint only occurs when the NameNode starts up. Work is in progress to support periodic checkpointing in the near future.

      (emphasis mine).

      Note that this directly conflicts with information in the hdfs user guide, http://hadoop.apache.org/common/docs/r0.20.2/hdfs_user_guide.html#Secondary+NameNode
      and http://hadoop.apache.org/hdfs/docs/current/hdfs_user_guide.html#Checkpoint+Node

      I haven't done a thorough audit of that doc-- I only noticed the above inaccuracy.

        Issue Links


          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.21.1 [ 12315271 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Fix Version/s 0.23.0 [ 12315571 ]
          Resolution Fixed [ 1 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Assignee Joe Crobak [ joecrobak ]
          Joe Crobak made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Joe Crobak made changes -
          Attachment HDFS-1612.patch [ 12471796 ]
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-1388 [ HDFS-1388 ]
          Joe Crobak created issue -


            • Assignee:
              Joe Crobak
              Joe Crobak
            • Votes:
              0 Vote for this issue
              3 Start watching this issue


              • Created: