Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8825 Enhancements to Balancer
  3. HDFS-8874

Add DN metrics for balancer and other block movement scenarios

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      For balancer, mover and migrator (HDFS-8789), we want to know how close it is to the DN's throttling thresholds. Although DN has existing metrics such as BytesWritten, BytesRead, CopyBlockOpNumOps and ReplaceBlockOpNumOps, there is no metrics to indicate the number of bytes moved.

      We can add ReplaceBlockBytesWritten and CopyBlockBytesRead to account for the bytes moved in ReplaceBlock and CopyBlock operations. In addition, we can also add throttling metrics for DataTransferThrottler and BlockBalanceThrottler.

        Attachments

        1. HDFS-8874-trunk.005.patch
          21 kB
          Chris Trezzo
        2. HDFS-8874-trunk.006.patch
          21 kB
          Chris Trezzo
        3. HDFS-8874-trunk-v1.patch
          9 kB
          Chris Trezzo
        4. HDFS-8874-trunk-v2.patch
          19 kB
          Chris Trezzo
        5. HDFS-8874-trunk-v3.patch
          17 kB
          Chris Trezzo
        6. HDFS-8874-trunk-v4.patch
          20 kB
          Chris Trezzo

          Issue Links

            Activity

              People

              • Assignee:
                ctrezzo Chris Trezzo
                Reporter:
                mingma Ming Ma
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: