Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15507

Add MapReduce counters about EC bytes read

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level ReadStatistics so from DFSClient we can know how much reads are EC/replication.

      In order for users to have a better view of how much of their workload is impacted by EC, we can expose EC read bytes to File System Counters, and to MapReduce's job counters. This way, end users can tell from MR jobs directly.

        Attachments

        1. HADOOP-15507.01.patch
          13 kB
          Xiao Chen
        2. image-2018-05-31-15-29-45-729.png
          174 kB
          Xiao Chen

          Activity

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: