Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5881

Create counters that count how many bytes of data were read from a local node and a local rack

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      MapReduce provides the counters "org.apache.hadoop.mapreduce.JobCounter DATA_LOCAL_MAPS" and "org.apache.hadoop.mapreduce.JobCounter RACK_LOCAL_MAPS" that count how many local map tasks were started. This works great, if we have one HDFS block per a input split.

      If we have multiple HDFS blocks in an input split (e.g. non-splittable file such as gzip, or use of CombineFileInputFormat), then these counters become inaccurate.

      If it has not been proposed yet, I propose adding new counters that count how many bytes of data were read from a local node and a local rack. Thanks to them, a user will see the real data locality ratio.

        Activity

        Adam Kawa made changes -
        Field Original Value New Value
        Description MapReduce provides the counter "org.apache.hadoop.mapreduce.JobCounter DATA_LOCAL_MAPS" and "org.apache.hadoop.mapreduce.JobCounter RACK_LOCAL_MAPS" that counts how many local map tasks were started. This works great, if we have one HDFS block per a input split.

        If we have multiple HDFS blocks in an input split (e.g. non-splittable file such as gzip, or use of CombineFileInputFormat), then this counter becomes inaccurate.

        If it has not been proposed yet, I propose adding new counters that counts how many bytes of data were read from a local node and a local rack. Thanks to them, a user will see the real data locality ratio.
        MapReduce provides the counters "org.apache.hadoop.mapreduce.JobCounter DATA_LOCAL_MAPS" and "org.apache.hadoop.mapreduce.JobCounter RACK_LOCAL_MAPS" that count how many local map tasks were started. This works great, if we have one HDFS block per a input split.

        If we have multiple HDFS blocks in an input split (e.g. non-splittable file such as gzip, or use of CombineFileInputFormat), then these counters become inaccurate.

        If it has not been proposed yet, I propose adding new counters that count how many bytes of data were read from a local node and a local rack. Thanks to them, a user will see the real data locality ratio.
        Adam Kawa created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Adam Kawa
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development