Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4116

Balancer should provide better resource management

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed DataNode protocol version without impact to clients other than to compel use of current version of client application.

      Description

      The number of threads are currently limited on datanodes. Once these threads are occupied, DataNode does not accept any more requests (DOS). Recently we saw a case where most of the 256 threads were waiting in DataXceiver.replaceBlock() trying to acquire balancingSem. Since rebalancing is (heavily) throttled, I would think this would be the common case.

      These operations waiting for active rebalancing threads to finish need not take up a thread.

        Attachments

        1. balancerRM.patch
          20 kB
          Hairong Kuang
        2. balancerRM1.patch
          21 kB
          Hairong Kuang
        3. balancerRM2.patch
          23 kB
          Hairong Kuang
        4. balancerRM2-b18.patch
          21 kB
          Hairong Kuang

          Activity

            People

            • Assignee:
              hairong Hairong Kuang
              Reporter:
              rangadi Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: