Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6658

Namenode memory optimization - Block replicas list

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.1
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      Part of the memory consumed by every BlockInfo object in the Namenode is a linked list of block references for every DatanodeStorageInfo (called "triplets").
      We propose to change the way we store the list in memory.
      Using primitive integer indexes instead of object references will reduce the memory needed for every block replica (when compressed oops is disabled) and in our new design the list overhead will be per DatanodeStorageInfo and not per block replica.

      see attached design doc. for details and evaluation results.

        Attachments

        1. New primative indexes.jpg
          93 kB
          Daryn Sharp
        2. Old triplets.jpg
          96 kB
          Daryn Sharp
        3. HDFS-6658.patch
          200 kB
          Daryn Sharp
        4. HDFS-6658.patch
          198 kB
          Daryn Sharp
        5. BlocksMap redesign.pdf
          73 kB
          Daryn Sharp
        6. HDFS-6658.patch
          79 kB
          Amir Langer
        7. BlockListOptimizationComparison.xlsx
          36 kB
          Amir Langer
        8. Namenode Memory Optimizations - Block replicas list.docx
          254 kB
          Amir Langer

          Issue Links

            Activity

              People

              • Assignee:
                daryn Daryn Sharp
                Reporter:
                langera Amir Langer
              • Votes:
                0 Vote for this issue
                Watchers:
                48 Start watching this issue

                Dates

                • Created:
                  Updated: