Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2362 More Improvements on NameNode Scalability
  3. HDFS-2476

More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha, 0.23.7
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This patch introduces two hash data structures for storing under-replicated, over-replicated and invalidated blocks.

      1. LightWeightHashSet
      2. LightWeightLinkedSet

      Currently in all these cases we are using java.util.TreeSet which adds unnecessary overhead.

      The main bottlenecks addressed by this patch are:
      -cluster instability times, when these queues (especially under-replicated) tend to grow quite drastically,
      -initial cluster startup, when the queues are initialized, after leaving safemode,
      -block reports,
      -explicit acks for block addition and deletion

      1. The introduced structures are CPU-optimized.
      2. They shrink and expand according to current capacity.
      3. Add/contains/delete ops are performed in O(1) time (unlike current log n for TreeSet).
      4. The sets are equipped with fast access methods for polling a number of elements (get+remove), which are used for handling the queues.

      1. hashStructures.patch-9
        70 kB
        Tomasz Nykiel
      2. hashStructures.patch-8
        70 kB
        Tomasz Nykiel
      3. hashStructures.patch-7
        69 kB
        Tomasz Nykiel
      4. hashStructures.patch-6
        69 kB
        Tomasz Nykiel
      5. hashStructures.patch-5
        69 kB
        Tomasz Nykiel
      6. hashStructures.patch-4
        69 kB
        Tomasz Nykiel
      7. hashStructures.patch-3
        69 kB
        Tomasz Nykiel
      8. hashStructures.patch-2
        69 kB
        Tomasz Nykiel
      9. hashStructures.patch
        69 kB
        Tomasz Nykiel

        Activity

        Kihwal Lee made changes -
        Fix Version/s 0.23.7 [ 12323955 ]
        Target Version/s 0.24.0, 0.23.3 [ 12317653, 12320052 ] 0.23.3, 0.24.0 [ 12320052, 12317653 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.0 [ 12320353 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.23.3 [ 12320052 ]
        Suresh Srinivas made changes -
        Fix Version/s 0.23.3 [ 12320052 ]
        Affects Version/s 0.23.0 [ 12315571 ]
        Target Version/s 0.24.0 [ 12317653 ] 0.24.0, 0.23.3 [ 12317653, 12320052 ]
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Target Version/s 0.24.0 [ 12317653 ]
        Fix Version/s 0.24.0 [ 12317653 ]
        Resolution Fixed [ 1 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-9 [ 12503471 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-8 [ 12502801 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-7 [ 12502566 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-6 [ 12500748 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-5 [ 12500315 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-4 [ 12500299 ]
        Tomasz Nykiel made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-3 [ 12500263 ]
        Tomasz Nykiel made changes -
        Attachment hashStructures.patch-2 [ 12499973 ]
        Tomasz Nykiel made changes -
        Field Original Value New Value
        Attachment hashStructures.patch [ 12499803 ]
        Tomasz Nykiel created issue -

          People

          • Assignee:
            Tomasz Nykiel
            Reporter:
            Tomasz Nykiel
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development