[MAPREDUCE-5881] Create counters that count how many bytes of data were read from a local node and a local rack

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: mrv2
• Assignee: Unassigned
• Reporter: Adam Kawa
• Created: 07/May/14 10:14
• Updated: 07/May/14 10:16

Description

MapReduce provides the counters "org.apache.hadoop.mapreduce.JobCounter DATA_LOCAL_MAPS" and "org.apache.hadoop.mapreduce.JobCounter RACK_LOCAL_MAPS" that count how many local map tasks were started. This works great, if we have one HDFS block per a input split.

If we have multiple HDFS blocks in an input split (e.g. non-splittable file such as gzip, or use of CombineFileInputFormat), then these counters become inaccurate.

If it has not been proposed yet, I propose adding new counters that count how many bytes of data were read from a local node and a local rack. Thanks to them, a user will see the real data locality ratio.