Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8825 Enhancements to Balancer
  3. HDFS-8874

Add DN metrics for balancer and other block movement scenarios

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      For balancer, mover and migrator (HDFS-8789), we want to know how close it is to the DN's throttling thresholds. Although DN has existing metrics such as BytesWritten, BytesRead, CopyBlockOpNumOps and ReplaceBlockOpNumOps, there is no metrics to indicate the number of bytes moved.

      We can add ReplaceBlockBytesWritten and CopyBlockBytesRead to account for the bytes moved in ReplaceBlock and CopyBlock operations. In addition, we can also add throttling metrics for DataTransferThrottler and BlockBalanceThrottler.

      Attachments

        1. HDFS-8874-trunk-v4.patch
          20 kB
          Chris Trezzo
        2. HDFS-8874-trunk-v3.patch
          17 kB
          Chris Trezzo
        3. HDFS-8874-trunk-v2.patch
          19 kB
          Chris Trezzo
        4. HDFS-8874-trunk-v1.patch
          9 kB
          Chris Trezzo
        5. HDFS-8874-trunk.006.patch
          21 kB
          Chris Trezzo
        6. HDFS-8874-trunk.005.patch
          21 kB
          Chris Trezzo

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-9028 Create accessor methods for DataNode#data and DataNode#isBlockTokenEnabled

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              ctrezzo Chris Trezzo
              mingma Ming Ma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: