Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4116

Balancer should provide better resource management

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.17.0
    • 0.19.0
    • None
    • None
    • Incompatible change, Reviewed
    • Changed DataNode protocol version without impact to clients other than to compel use of current version of client application.

    Description

      The number of threads are currently limited on datanodes. Once these threads are occupied, DataNode does not accept any more requests (DOS). Recently we saw a case where most of the 256 threads were waiting in DataXceiver.replaceBlock() trying to acquire balancingSem. Since rebalancing is (heavily) throttled, I would think this would be the common case.

      These operations waiting for active rebalancing threads to finish need not take up a thread.

      Attachments

        1. balancerRM2-b18.patch
          21 kB
          Hairong Kuang
        2. balancerRM2.patch
          23 kB
          Hairong Kuang
        3. balancerRM1.patch
          21 kB
          Hairong Kuang
        4. balancerRM.patch
          20 kB
          Hairong Kuang

        Activity

          People

            hairong Hairong Kuang
            rangadi Raghu Angadi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: