Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12288

Fix DataNode's xceiver count calculation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.1, 3.4.0
    • datanode, hdfs
    • None
    • Reviewed

    Description

      The problem with the ThreadGroup.activeCount() method is that the method is only a very rough estimate, and in reality returns the total number of threads in the thread group as opposed to the threads actually running.

      In some DNs, we saw this to return 50~ for a long time, even though the actual number of DataXceiver threads was next to none.
      This is a big issue as we use the xceiverCount to make decisions on the NN for choosing replication source DN or returning DNs to clients for R/W.

      The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value which only accounts for actual number of DataXcevier threads currently running and thus represents the load on the DN much better.

      Attachments

        1. HDFS-12288.001.patch
          6 kB
          Lukas Majercak
        2. HDFS-12288.002.patch
          6 kB
          Lukas Majercak
        3. HDFS-12288.003.patch
          8 kB
          Chen Zhang
        4. HDFS-12288.004.patch
          11 kB
          Chen Zhang
        5. HDFS-12288.005.patch
          13 kB
          Chen Zhang
        6. HDFS-12288.006.patch
          13 kB
          Chen Zhang
        7. HDFS-12288.007.patch
          13 kB
          Chen Zhang
        8. HDFS-12288.008.patch
          14 kB
          Lisheng Sun

        Issue Links

          Activity

            People

              leosun08 Lisheng Sun
              lukmajercak Lukas Majercak
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: