[HADOOP-4116] Balancer should provide better resource management

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.17.0
• Fix Version/s: 0.19.0
• Hadoop Flags: Incompatible change, Reviewed
• Release Note: Changed DataNode protocol version without impact to clients other than to compel use of current version of client application.
• Assignee: Hairong Kuang
• Reporter: Raghu Angadi
• Created: 08/Sep/08 16:23
• Updated: 08/Jul/09 16:43
• Resolved: 25/Sep/08 20:28

Description

The number of threads are currently limited on datanodes. Once these threads are occupied, DataNode does not accept any more requests (DOS). Recently we saw a case where most of the 256 threads were waiting in DataXceiver.replaceBlock() trying to acquire balancingSem. Since rebalancing is (heavily) throttled, I would think this would be the common case.

These operations waiting for active rebalancing threads to finish need not take up a thread.