Hadoop Common
  1. Hadoop Common
  2. HADOOP-4116

Balancer should provide better resource management

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed DataNode protocol version without impact to clients other than to compel use of current version of client application.

      Description

      The number of threads are currently limited on datanodes. Once these threads are occupied, DataNode does not accept any more requests (DOS). Recently we saw a case where most of the 256 threads were waiting in DataXceiver.replaceBlock() trying to acquire balancingSem. Since rebalancing is (heavily) throttled, I would think this would be the common case.

      These operations waiting for active rebalancing threads to finish need not take up a thread.

      1. balancerRM2.patch
        23 kB
        Hairong Kuang
      2. balancerRM2-b18.patch
        21 kB
        Hairong Kuang
      3. balancerRM1.patch
        21 kB
        Hairong Kuang
      4. balancerRM.patch
        20 kB
        Hairong Kuang

        Activity

        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]
        Raghu Angadi made changes -
        Fix Version/s 0.18.2 [ 12313424 ]
        Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
        Fix Version/s 0.19.0 [ 12313211 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Nigel Daley made changes -
        Fix Version/s 0.19.0 [ 12313211 ]
        Robert Chansler made changes -
        Release Note Changed DataNode protocol version without impact to clients other than to compel use of current version of client application.
        Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
        Hairong Kuang made changes -
        Hadoop Flags [Reviewed] [Incompatible change, Reviewed]
        Hairong Kuang made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hairong Kuang made changes -
        Hadoop Flags [Reviewed]
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hairong Kuang made changes -
        Attachment balancerRM2.patch [ 12390875 ]
        Hairong Kuang made changes -
        Attachment balancerRM-b18.patch [ 12390869 ]
        Hairong Kuang made changes -
        Attachment balancerRM2-b18.patch [ 12390873 ]
        Hairong Kuang made changes -
        Attachment balancerRM-b18.patch [ 12390869 ]
        Hairong Kuang made changes -
        Attachment balancerRM1.patch [ 12390768 ]
        Hairong Kuang made changes -
        Attachment balancerRM1.patch [ 12390802 ]
        Hairong Kuang made changes -
        Attachment balancerRM1.patch [ 12390768 ]
        Hairong Kuang made changes -
        Issue Type Improvement [ 4 ] Bug [ 1 ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Fix Version/s 0.19.0 [ 12313211 ]
        Fix Version/s 0.18.2 [ 12313424 ]
        Hairong Kuang made changes -
        Attachment balancerRM.patch [ 12390414 ]
        Hairong Kuang made changes -
        Summary DataNode : idle rebalancing operations need not take up threads. Balancer should provide better resource management
        Assignee Hairong Kuang [ hairong ]
        Raghu Angadi made changes -
        Field Original Value New Value
        Description
        The number of threads are currently limited on datanodes. Once these threads are occupied, DataNode does not accept any more requests (DOS). Recently we saw a case where most of the 256 threads were waiting in {{DataXceiver.replaceBlock()}} trying to acquire {{balancingSem}}. Since rebalancing is (heavily) throttled, I would think this would be the common case.

        These waiting threads need not take up a thread.
        The number of threads are currently limited on datanodes. Once these threads are occupied, DataNode does not accept any more requests (DOS). Recently we saw a case where most of the 256 threads were waiting in {{DataXceiver.replaceBlock()}} trying to acquire {{balancingSem}}. Since rebalancing is (heavily) throttled, I would think this would be the common case.

        These operations waiting for active rebalancing threads to finish need not take up a thread.
        Summary DataNode : idle rebalancing threads need not take up threads. DataNode : idle rebalancing operations need not take up threads.
        Raghu Angadi created issue -

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Raghu Angadi
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development