[HDFS-1172] Blocks in newly completed files are considered under-replicated too quickly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.21.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Masatake Iwasaki Assign to me
• Reporter: Todd Lipcon
• Created: 22/May/10 06:20
• Updated: 30/Aug/16 01:43
• Resolved: 14/Oct/15 06:09

Description

I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.

Issue Links

• is duplicated by: HDFS-8763 After file closed, a race condition between IBR of 3rd replica of lastBlock and ReplicationMonitor
• relates to: HDFS-8999 Allow a file to be closed with COMMITTED but not yet COMPLETE blocks.
• relates to: HDFS-9535 Newly completed blocks in IBR should not be considered under-replicated too quickly