[MAPREDUCE-2025] Improve job counter wire protocol with incremental schema and delta values

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.20.2
• Component/s: client, jobtracker, task
• Tags: counter
• Assignee: Luke Lu
• Reporter: Luke Lu
• Created: 21/Aug/10 01:32
• Updated: 13/Jan/11 02:29

Description

The current job counter protocol is quite verbose (string key and vlong value pairs) especially when keys are long. By using an incremental schema (key to key id mapping) and data (vint key id to vlong value delta) protocol, we can significantly lower the counter overhead and allow us to up (from the current 1 update/minute) the counter update frequency. This jira tracks the user counter (for apps or tools (pig etc.) built on top of mr framework) improvement. Framework (mr/hdfs etc) counter improvement is tracked by MAPREDUCE-901 (a more common case with more optimization opportunities, as the schema is fixed.)

Issue Links

• is related to: MAPREDUCE-901 Move Framework Counters into a TaskMetric structure