[HADOOP-1187] DFS Scalability: avoid scanning entire list of datanodes in getAdditionalBlocks

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Dhruba Borthakur
• Reporter: Dhruba Borthakur
• Created: 30/Mar/07 21:32
• Updated: 08/Jul/09 16:42
• Resolved: 04/Apr/07 19:46

Description

A new block allocations for a file scans the list of all known datanodes to find if the client that is a also a cluster node. If so, then it tries to allocate a replica locally. This check consumes plenty of CPU, especially if the number of datanodes in a cluster is large.

An optimization: if the client is also a cluster node, then cache a reference to the corresponding DatanodeDescriptor from the entry in pendingCreate. The method getAdditionalBlock() uses the cached DatanodeDescriptor and thus avoids scanning the entire list of datanodes.