[HADOOP-14412] HostsFileReader#getHostDetails is very expensive on large clusters

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
• Component/s: util
• Target Version/s: 2.8.2
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 11/May/17 15:52
• Updated: 24/Apr/18 20:50
• Resolved: 17/May/17 03:02

Description

After upgrading one of our large clusters to 2.8 we noticed many IPC server threads of the resourcemanager spending time in NodesListManager#isValidNode which in turn was calling HostsFileReader#getHostDetails. The latter is creating complete copies of the include and exclude sets for every node heartbeat, and these sets are not small due to the size of the cluster. These copies are causing multiple resizes of the underlying HashSets being filled and creating lots of garbage.

Issue Links

• is broken by: YARN-4002 make ResourceTrackerService.nodeHeartbeat more concurrent