Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2126 Improve Namenode startup time [umbrella task]
  3. HDFS-1070

Speedup NameNode image loading and saving by storing local file names

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.23.0
    • namenode
    • None
    • Reviewed
    • Hide
      This changes the fsimage format to be
      root directory-1 directory-2 ... directoy-n.
      Each directory stores all its children in the following format:
      Directory_full_path_name num_of_children child-1 ... child-n.
      Each inode stores only the last component of its path name into fsimage.
      This change requires an upgrade at deployment.
      Show
      This changes the fsimage format to be root directory-1 directory-2 ... directoy-n. Each directory stores all its children in the following format: Directory_full_path_name num_of_children child-1 ... child-n. Each inode stores only the last component of its path name into fsimage. This change requires an upgrade at deployment.

    Description

      Currently each inode stores its full path in the fsimage. I'd propose to store the local name instead. In order for each inode to identify its parent, all inodes in a directory tree are stored in the image in in-order. This proposal also requires each directory stores the number of its children in image.

      This proposal would bring a few benefits as pointed below and therefore speedup the image loading and saving.

      1. Remove the overhead of converting java-UTF8 encoded local name to string-represented full path then to UTF8 encoded full path when saving to an image and vice versa when loading the image.
      2. Remove the overhead of traversing the full path when inserting the inode to its parent inode.
      3. Reduce the number of temporary java objects during the process of image loading or saving and therefore reduce the GC overhead.
      4. Reduce the size of an image.

      Attachments

        1. trunkLocalNameImage9.patch
          14 kB
          Hairong Kuang
        2. trunkLocalNameImage9.patch
          14 kB
          Hairong Kuang
        3. trunkLocalNameImage8.patch
          14 kB
          Hairong Kuang
        4. trunkLocalNameImage7.patch
          13 kB
          Hairong Kuang
        5. trunkLocalNameImage6.patch
          12 kB
          Hairong Kuang
        6. trunkLocalNameImage5.patch
          12 kB
          Hairong Kuang
        7. trunkLocalNameImage4.patch
          12 kB
          Hairong Kuang
        8. trunkLocalNameImage3.patch
          33 kB
          Hairong Kuang
        9. trunkLocalNameImage1.patch
          27 kB
          Hairong Kuang
        10. trunkLocalNameImage.patch
          21 kB
          Hairong Kuang

        Issue Links

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              1 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: