Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9579

Provide bytes-read-by-network-distance metrics at FileSystem.Statistics level



    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Hadoop Flags:


      For cross DC distcp or other applications, it becomes useful to have insight as to the traffic volume for each network distance to distinguish cross-DC traffic, local-DC-remote-rack, etc.

      FileSystem's existing bytesRead metrics tracks all the bytes read. To provide additional metrics for each network distance, we can add additional metrics to FileSystem level and have DFSInputStream update the value based on the network distance between client and the datanode.

      DFSClient will resolve client machine's network location as part of its initialization. It doesn't need to resolve datanode's network location for each read as DatanodeInfo already has the info.

      There are existing HDFS specific metrics such as ReadStatistics and DFSHedgedReadMetrics. But these metrics are only accessible via DFSClient or DFSInputStream. Not something that application framework such as MR and Tez can get to. That is the benefit of storing these new metrics in FileSystem.Statistics.

      This jira only includes metrics generation by HDFS. The consumption of these metrics at MR and Tez will be tracked by separated jiras.

      We can add similar metrics for HDFS write scenario later if it is necessary.


        1. HDFS-9579.patch
          15 kB
          Ming Ma
        2. HDFS-9579-2.patch
          16 kB
          Ming Ma
        3. HDFS-9579-3.patch
          17 kB
          Ming Ma
        4. MR job counters.png
          133 kB
          Ming Ma
        5. HDFS-9579-4.patch
          17 kB
          Ming Ma
        6. HDFS-9579-5.patch
          19 kB
          Ming Ma
        7. HDFS-9579-6.patch
          19 kB
          Ming Ma
        8. HDFS-9579-7.patch
          39 kB
          Ming Ma
        9. HDFS-9579-8.patch
          41 kB
          Ming Ma
        10. HDFS-9579-9.patch
          42 kB
          Ming Ma
        11. HDFS-9579-10.patch
          42 kB
          Ming Ma
        12. HDFS-9579-branch-2.patch
          40 kB
          Ming Ma

          Issue Links



              • Assignee:
                mingma Ming Ma
                mingma Ming Ma
              • Votes:
                0 Vote for this issue
                13 Start watching this issue


                • Created: