[HDFS-9783] DataNode deadlocks after running hdfs mover

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.6.0
• Component/s: datanode
• Assignee: Unassigned
• Reporter: David Watzke
• Created: 09/Feb/16 15:28
• Updated: 09/Feb/16 16:18
• Resolved: 09/Feb/16 16:18

Description

We're running CDH 5.4.4 (with Hadoop 2.6.0). Recently we've added 800T of ARCHIVE storage, marked some data (16T * 2 repl. factor) as "COLD" and ran the hdfs mover to enforce the storage policy.

After a few minutes, the mover hangs because one or more datanodes hang as well. Please check out the deadlock revealed by jstack. Also, here's what appeared in DN log:

2016-02-09 15:58:15,676 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Not able to copy block 1157230686 to /5.45.60.142:40144 because threads quota is exceeded.

Issue Links

• is duplicated by: HDFS-9661 Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and createRbw