[TEZ-3911] Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Assignee: Vineet Garg
• Reporter: Eric Wohlstadter
• Due: 30/Apr/18
• Created: 05/Apr/18 18:40
• Updated: 09/May/18 19:04
• Resolved: 09/May/18 19:04

Description

Consumers of HistoryLoggingService reported counters are currently required to compute any task-level aggregations other than "sum". This is inefficient as Tez is already "scanning" over this data. Computing incremental aggregates shouldn't require additional scans by ATS consumers.

Provide an option for Task counter aggregations other than "sum". Computation of these extra counters can be turned on/off.

The option will generate "synthetic" counters at final aggregation time for reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS.

Only incremental aggregations will be supported (min/max/avg). Aggregation computation will be folded into the existing "aggregation loop" beginning at VertexImpl.incrTaskCounters.

Extra aggregations will only be supported during final counter aggregation.

Aggregations will only include the "bestAttempt" for each task.

A design doc will be provided.

Because final task aggregation holds a lock, a performance report will be provided.