[HDFS-6709] Implement off-heap data structures for NameNode and other HDFS memory optimization

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: namenode
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 18/Jul/14 23:54
• Updated: 14/Oct/14 11:35

Description

We should investigate implementing off-heap data structures for NameNode and other HDFS memory optimization. These data structures could reduce latency by avoiding the long GC times that occur with large Java heaps. We could also avoid per-object memory overheads and control memory layout a little bit better. This also would allow us to use the JVM's "compressed oops" optimization even with really large namespaces, if we could get the Java heap below 32 GB for those cases. This would provide another performance and memory efficiency boost.

Issue Links

• is related to: HDFS-7244 Reduce Namenode memory using Flyweight pattern
• relates to: HBASE-10655 ByteRange-fronted slab allocator for on- and off-heap cellblock storage
• relates to: HBASE-10191 Move large arena storage off heap