[HDFS-7217] Better batching of IBRs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 08/Oct/14 23:29
• Updated: 01/Dec/14 03:11
• Resolved: 09/Oct/14 17:12

Description

After HDFS-2691 (pipeline recovery in HA), the number of IBR(incremental block report)s have doubled. Since processing of IBR requires exclusive FSNamesystem write lock, this can be a source of significant overhead on clusters with high write load.

On one of the busy clusters, we have observed 60 to 70 percent of available handlers being constantly occupied by IBRs. This degrades throughput greatly when compared to 0.23.

Issue Links

• breaks: HDFS-7226 TestDNFencing.testQueueingWithAppend failed often in latest test