[HDFS-6599] 2.4 addBlock is 10 to 20 times slower compared to 0.23

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.4.0
• Fix Version/s: 2.5.0
• Target Version/s: 2.5.0
• Hadoop Flags: Reviewed
• Assignee: Daryn Sharp
• Reporter: Kihwal Lee
• Created: 24/Jun/14 18:20
• Updated: 20/Mar/15 18:19
• Resolved: 18/Jul/14 18:05

Description

From one of our busiest 0.23 clusters:

"AddBlockAvgTime" : 0.9514711501719515
"CreateAvgTime" : 1.7564162389174
"CompleteAvgTime" : 1.3310406035056548
"BlockReceivedAndDeletedAvgTime" : 0.661210005151392

From a not-so-busy 2.4 cluster:

"AddBlockAvgTime" : 10.083333333333334
"CreateAvgTime" : 1.0
"CompleteAvgTime" : 1.1111111111111112
"BlockReceivedAndDeletedAvgTime" : 0.07692307692307694

When the 2.4 cluster gets a moderate amount of write requests, the latency is terrible. E.g. addBlock goes upward of 60ms.

Issue Links

• is broken by: HDFS-5837 dfs.namenode.replication.considerLoad does not consider decommissioned nodes