Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5881

Create counters that count how many bytes of data were read from a local node and a local rack

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      MapReduce provides the counters "org.apache.hadoop.mapreduce.JobCounter DATA_LOCAL_MAPS" and "org.apache.hadoop.mapreduce.JobCounter RACK_LOCAL_MAPS" that count how many local map tasks were started. This works great, if we have one HDFS block per a input split.

      If we have multiple HDFS blocks in an input split (e.g. non-splittable file such as gzip, or use of CombineFileInputFormat), then these counters become inaccurate.

      If it has not been proposed yet, I propose adding new counters that count how many bytes of data were read from a local node and a local rack. Thanks to them, a user will see the real data locality ratio.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kawaa Adam Kawa
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: