Hadoop Common
  1. Hadoop Common
  2. HADOOP-4116

Balancer should provide better resource management

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed DataNode protocol version without impact to clients other than to compel use of current version of client application.

      Description

      The number of threads are currently limited on datanodes. Once these threads are occupied, DataNode does not accept any more requests (DOS). Recently we saw a case where most of the 256 threads were waiting in DataXceiver.replaceBlock() trying to acquire balancingSem. Since rebalancing is (heavily) throttled, I would think this would be the common case.

      These operations waiting for active rebalancing threads to finish need not take up a thread.

      1. balancerRM.patch
        20 kB
        Hairong Kuang
      2. balancerRM1.patch
        21 kB
        Hairong Kuang
      3. balancerRM2.patch
        23 kB
        Hairong Kuang
      4. balancerRM2-b18.patch
        21 kB
        Hairong Kuang

        Activity

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Raghu Angadi
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development