[HDFS-15907] Reduce Memory Overhead of AclFeature by avoiding AtomicInteger

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: namenode
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 19/Mar/21 16:36
• Updated: 27/Jan/24 02:50
• Resolved: 23/Mar/21 20:53

Description

In HDFS-15792 we made some changes to the AclFeature and ReferenceCountedMap classes to address a rare bug when loading the FSImage in parallel.

One change we made was to replace an int inside AclFeature with an AtomicInteger to avoid synchronising the methods in AclFeature.

Discussing this change with weichiu, he pointed out that while the AclFeature cache is intended to reduce the count of AclFeature objects, on a large cluster, it is possible for there to be many millions of AclFeature objects.

Previously, the int will have taken 4 bytes of heap.

By moving to a AtomicInteger, we probably have an overhead of:

4 bytes (or 8 if the heap is over 32GB) for a reference to the atomic long object
12 byte overhead for the java object
4 bytes inside the atomic long to store an int.

So the total heap overhead has gone from 4 bytes to 20 bytes just to use an AtomicInteger.

Therefore I think it makes sense to remove the AtomicInteger and just synchronise the methods of AclFeature where the value is incremented / decremented / retrieved.

Issue Links

• relates to: HDFS-15792 ClasscastException while loading FSImage