[HDFS-7461] Reduce impact of laggards on Mover

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: balancer & mover
• Assignee: Benoy Antony
• Reporter: Benoy Antony
• Created: 02/Dec/14 02:47
• Updated: 18/Dec/14 01:19

Description

The current Mover logic is as follows :

 for (Path target : targetPaths) {
        hasRemaining |= processPath(target.toUri().getPath());
      }
// wait for pending move to finish and retry the failed migration
hasRemaining |= Dispatcher.waitForMoveCompletion(storages.targets.values());

The processPath will schedule moves, but it is bounded by the number of concurrent moves (default is 5 per node} . Once block moves are scheduled, it will wait for ALL scheduled moves to finish in waitForMoveCompletion.
One slow move could keep the Mover idle for a long time.

It will be a performance improvement to schedule the next moves as soon as any (source , target) slot is available instead of waiting for all the scheduled moves to finish.