Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This changes the fsimage format to be
      root directory-1 directory-2 ... directoy-n.
      Each directory stores all its children in the following format:
      Directory_full_path_name num_of_children child-1 ... child-n.
      Each inode stores only the last component of its path name into fsimage.
      This change requires an upgrade at deployment.
      Show
      This changes the fsimage format to be root directory-1 directory-2 ... directoy-n. Each directory stores all its children in the following format: Directory_full_path_name num_of_children child-1 ... child-n. Each inode stores only the last component of its path name into fsimage. This change requires an upgrade at deployment.

      Description

      Currently each inode stores its full path in the fsimage. I'd propose to store the local name instead. In order for each inode to identify its parent, all inodes in a directory tree are stored in the image in in-order. This proposal also requires each directory stores the number of its children in image.

      This proposal would bring a few benefits as pointed below and therefore speedup the image loading and saving.

      1. Remove the overhead of converting java-UTF8 encoded local name to string-represented full path then to UTF8 encoded full path when saving to an image and vice versa when loading the image.
      2. Remove the overhead of traversing the full path when inserting the inode to its parent inode.
      3. Reduce the number of temporary java objects during the process of image loading or saving and therefore reduce the GC overhead.
      4. Reduce the size of an image.

        Attachments

        1. trunkLocalNameImage.patch
          21 kB
          Hairong Kuang
        2. trunkLocalNameImage1.patch
          27 kB
          Hairong Kuang
        3. trunkLocalNameImage3.patch
          33 kB
          Hairong Kuang
        4. trunkLocalNameImage4.patch
          12 kB
          Hairong Kuang
        5. trunkLocalNameImage5.patch
          12 kB
          Hairong Kuang
        6. trunkLocalNameImage6.patch
          12 kB
          Hairong Kuang
        7. trunkLocalNameImage7.patch
          13 kB
          Hairong Kuang
        8. trunkLocalNameImage8.patch
          14 kB
          Hairong Kuang
        9. trunkLocalNameImage9.patch
          14 kB
          Hairong Kuang
        10. trunkLocalNameImage9.patch
          14 kB
          Hairong Kuang

          Issue Links

            Activity

              People

              • Assignee:
                hairong Hairong Kuang
                Reporter:
                hairong Hairong Kuang
              • Votes:
                1 Vote for this issue
                Watchers:
                28 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: