Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5145

Balancer sometimes runs out of memory after days or weeks running

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The culprit is a HashMap called MovedBlocks. By design this map does not get cleaned up between iterations. This is because the deletion of source replicas is done by NN. When next iteration starts, source replicas may not have been deleted, Balancer does not want to schedule them to move again. To prevent running out of memory, Balancer should expire/clean the movedBlocks from some iterations back.

        Attachments

        1. balancerOME.patch
          6 kB
          Hairong Kuang
        2. balancerOME1.patch
          6 kB
          Hairong Kuang

          Activity

            People

            • Assignee:
              hairong Hairong Kuang
              Reporter:
              hairong Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: