Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This changes the fsimage format to be
      root directory-1 directory-2 ... directoy-n.
      Each directory stores all its children in the following format:
      Directory_full_path_name num_of_children child-1 ... child-n.
      Each inode stores only the last component of its path name into fsimage.
      This change requires an upgrade at deployment.
      Show
      This changes the fsimage format to be root directory-1 directory-2 ... directoy-n. Each directory stores all its children in the following format: Directory_full_path_name num_of_children child-1 ... child-n. Each inode stores only the last component of its path name into fsimage. This change requires an upgrade at deployment.

      Description

      Currently each inode stores its full path in the fsimage. I'd propose to store the local name instead. In order for each inode to identify its parent, all inodes in a directory tree are stored in the image in in-order. This proposal also requires each directory stores the number of its children in image.

      This proposal would bring a few benefits as pointed below and therefore speedup the image loading and saving.

      1. Remove the overhead of converting java-UTF8 encoded local name to string-represented full path then to UTF8 encoded full path when saving to an image and vice versa when loading the image.
      2. Remove the overhead of traversing the full path when inserting the inode to its parent inode.
      3. Reduce the number of temporary java objects during the process of image loading or saving and therefore reduce the GC overhead.
      4. Reduce the size of an image.
      1. trunkLocalNameImage.patch
        21 kB
        Hairong Kuang
      2. trunkLocalNameImage1.patch
        27 kB
        Hairong Kuang
      3. trunkLocalNameImage3.patch
        33 kB
        Hairong Kuang
      4. trunkLocalNameImage4.patch
        12 kB
        Hairong Kuang
      5. trunkLocalNameImage5.patch
        12 kB
        Hairong Kuang
      6. trunkLocalNameImage6.patch
        12 kB
        Hairong Kuang
      7. trunkLocalNameImage7.patch
        13 kB
        Hairong Kuang
      8. trunkLocalNameImage8.patch
        14 kB
        Hairong Kuang
      9. trunkLocalNameImage9.patch
        14 kB
        Hairong Kuang
      10. trunkLocalNameImage9.patch
        14 kB
        Hairong Kuang

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              1 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development