Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15907

Reduce Memory Overhead of AclFeature by avoiding AtomicInteger



    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.1, 3.4.0
    • 3.3.1, 3.4.0
    • namenode
    • None
    • Reviewed


      In HDFS-15792 we made some changes to the AclFeature and ReferenceCountedMap classes to address a rare bug when loading the FSImage in parallel.

      One change we made was to replace an int inside AclFeature with an AtomicInteger to avoid synchronising the methods in AclFeature.

      Discussing this change with weichiu, he pointed out that while the AclFeature cache is intended to reduce the count of AclFeature objects, on a large cluster, it is possible for there to be many millions of AclFeature objects.

      Previously, the int will have taken 4 bytes of heap.

      By moving to a AtomicInteger, we probably have an overhead of:

      4 bytes (or 8 if the heap is over 32GB) for a reference to the atomic long object
      12 byte overhead for the java object
      4 bytes inside the atomic long to store an int.

      So the total heap overhead has gone from 4 bytes to 20 bytes just to use an AtomicInteger.

      Therefore I think it makes sense to remove the AtomicInteger and just synchronise the methods of AclFeature where the value is incremented / decremented / retrieved.


        1. HDFS-15907.001.patch
          2 kB
          Stephen O'Donnell

        Issue Links



              sodonnell Stephen O'Donnell
              sodonnell Stephen O'Donnell
              0 Vote for this issue
              7 Start watching this issue