[HDFS-11661] GetContentSummary uses excessive amounts of memory

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.0.0-alpha2
• Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
• Component/s: namenode
• Target Version/s: 3.0.0-alpha4, 2.8.1
• Hadoop Flags: Reviewed

Release Note

Reverted HDFS-10797 to fix a scalability regression brought by the commit.

• Assignee: Wei-Chiu Chuang
• Reporter: Nathan Roberts
• Created: 17/Apr/17 21:23
• Updated: 25/Oct/19 20:24
• Resolved: 25/May/17 01:23

Description

ContentSummaryComputationContext::nodeIncluded() is being used to keep track of all INodes visited during the current content summary calculation. This can be all of the INodes in the filesystem, making for a VERY large hash table. This simply won't work on large filesystems.

We noticed this after upgrading a namenode with ~100Million filesystem objects was spending significantly more time in GC. Fortunately this system had some memory breathing room, other clusters we have will not run with this additional demand on memory.

This was added as part of HDFS-10797 as a way of keeping track of INodes that have already been accounted for - to avoid double counting.

Issue Links

• is broken by: HDFS-10797 Disk usage summary of snapshots causes renamed blocks to get counted twice
• relates to: HDFS-11515 -du throws ConcurrentModificationException