[HDFS-10966] Enhance Dispatcher logic on deciding when to give up a source DataNode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7.4, 3.0.0-alpha2
• Component/s: balancer & mover
• Target Version/s: 2.7.4
• Hadoop Flags: Reviewed
• Assignee: Mark Wagner
• Reporter: Zhe Zhang
• Created: 05/Oct/16 22:41
• Updated: 21/Nov/16 23:02
• Resolved: 21/Nov/16 23:00

Description

When a Dispatcher thread works on a source DataNode, in each iteration it tries to execute a PendingMove. If no block is moved after 5 iterations, this source (over-utlized) DataNode is given up for this Balancer iteration (20 mins). This is problematic if the source DataNode was heavily loaded in the beginning of the iteration. It will quickly encounter 5 unsuccessful moves and be abandoned.

We should enhance this logic by e.g. using elapsed time instead of number of iterations.

// Check if the previous move was successful
        } else {
          // source node cannot find a pending block to move, iteration +1
          noPendingMoveIteration++;
          // in case no blocks can be moved for source node's task,
          // jump out of while-loop after 5 iterations.
          if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
            LOG.info("Failed to find a pending move "  + noPendingMoveIteration
                + " times.  Skipping " + this);
            resetScheduledSize();
          }
        }

Issue Links

• relates to: HDFS-11015 Enforce timeout in balancer