[FLINK-456] Integrate runtime metrics / statistics

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Component/s: Runtime / Coordination, Runtime / Metrics
• Labels: github-import
• Assignee: Robert Metzger
• Reporter: Fabian Hueske
• Created: 09/Jun/14 12:19
• Updated: 28/Feb/19 14:01
• Resolved: 26/Feb/19 15:19

Description

The engine should collect job execution statistics (e.g., via accumulators) such as:

• total number of input / output records per operator
• histogram of input/output ratio of UDF calls
• histogram of number of input records per reduce / cogroup UDF call
• histogram of number of output records per UDF call
• histogram of time spend in UDF calls
• number of local and remote bytes read (not via accumulators)
• ...

These stats should be made available to the user after execution (via webfrontend). The purpose of this feature is to ease performance debugging of parallel jobs (e.g., to detect data skew).
It should be possible to deactivate (or activate) the gathering of these statistics.

---------------- Imported from GitHub ----------------
Url: https://github.com/stratosphere/stratosphere/issues/456
Created by: fhueske
Labels: enhancement, runtime, user satisfaction,
Created at: Tue Feb 04 20:32:49 CET 2014
State: open

Issue Links

• relates to: FLINK-1297 Add support for tracking statistics of intermediate results

Sub-Tasks

• Integrate metrics library and report basic metrics to JobManager web interface: Resolved
• Expose metrics to graphite, ganglia and JMX.: Closed
• Show JVM Metrics for JobManager: Closed
• Add per-job metrics to flink.: Resolved
• Improve TM Monitoring: CPU utilization, hide graphs by default and show summary only: Resolved
• Report accumulators periodically while job is running: Closed
• Expose task accumulators via JMX: Closed
• Measure latency of elements and expose it as a metric: Resolved
• Collect Metrics in Runtime Operators: Closed
• Add Meter Metric Type: Closed
• Add Histogram Metric Type: Closed
• Expose JobManager Metrics: Closed
• Expose metrics to Webfrontend: Closed
• Show metrics in WebFrontend: Resolved