[HADOOP-492] Global counters

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Assignee: David Bowen
• Reporter: arkady borkovsky
• Created: 29/Aug/06 21:34
• Updated: 08/Jul/09 16:51
• Resolved: 23/Feb/07 20:13

Description

It would be nice to have map / reduce job keep aggregated counts for arbitrary events occuring in its tasks – the numer of records processed, the numer of exceptions of a specific type, the number of sentences in passive voice, whatever the jobs finds useful.

This can be implemented by tasks periodically sending <name, value> pairs to the jobtracker (in some implementations such messages are piggy-backed on the heartbeats), so that the job tracker stores all the latests values from each task and aggregates them on a request. It should also make the aggregated values available at the job end. The value for a task would be flushed when the task fails.

#491 and #490 may be related to this one.

Issue Links

• is related to: HADOOP-481 Hadoop mapred metrics should include per job input/output statistics rather than per-task statistics