[HDFS-13174] hdfs mover -p /path times out after 20 min

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
• Fix Version/s: 3.2.0, 3.1.1, 3.0.4, 2.10.2
• Component/s: balancer & mover
• Target Version/s: 2.7.8

Release Note

Mover could have fail after 20+ minutes if a block move was enqueued for this long, between two DataNodes due to an internal constant that was introduced for Balancer, but affected Mover as well.
The internal constant can be configured with the dfs.balancer.max-iteration-time parameter after the patch, and affects only the Balancer. Default is 20 minutes.

• Assignee: István Fajth
• Reporter: István Fajth
• Created: 20/Feb/18 13:35
• Updated: 17/Mar/21 04:14
• Resolved: 15/Jun/18 20:46

Description

In HDFS-11015 there is an iteration timeout introduced in Dispatcher.Source class, that is checked during dispatching the moves that the Balancer and the Mover does. This timeout is hardwired to 20 minutes.

In the Balancer we have iterations, and even if an iteration is timing out the Balancer runs further and does an other iteration before it fails if there were no moves happened in a few iterations.

The Mover on the other hand does not have iterations, so if moving a path runs for more than 20 minutes, and there are moves decided and enqueued between two DataNode, after 20 minutes Mover will stop with the following exception reported to the console (lines might differ as this exception came from a CDH5.12.1 installation).
java.io.IOException: Block move timed out
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:382)
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:328)
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2500(Dispatcher.java:186)
at org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:956)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

Note that this issue is not coming up if all blocks can be moved inside the DataNodes without having to move the block to an other DataNode.

Issue Links

• causes: HDFS-13975 TestBalancer#testMaxIterationTime fails sporadically
• is caused by: HDFS-11015 Enforce timeout in balancer