[YARN-3816] [Aggregation] App-level aggregation and accumulation for YARN system metrics

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: timelineserver
• Labels: yarn-2928-1st-milestone
• Target Version/s: YARN-2928
• Hadoop Flags: Reviewed
• Assignee: Li Lu
• Reporter: Junping Du
• Created: 17/Jun/15 14:56
• Updated: 25/Oct/19 20:27
• Resolved: 22/Apr/16 17:27

Description

We need application level aggregation of Timeline data:

• To present end user aggregated states for each application, include: resource (CPU, Memory) consumption across all containers, number of containers launched/completed/failed, etc. We need this for apps while they are running as well as when they are done.
• Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be aggregated to show details of states in framework level.
• Other level (Flow/User/Queue) aggregation can be more efficient to be based on Application-level aggregations rather than raw entity-level data as much less raws need to scan (with filter out non-aggregated entities, like: events, configurations, etc.).

Issue Links

• is related to: YARN-7272 Enable timeline collector fault tolerance
• is related to: YARN-5747 Application timeline metric aggregation in timeline v2 will lose last round aggregation when an application finishes