[REEF-1732] Build Metrics System

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: IMRU, REEF
• Assignee: Julia Wang
• Reporter: Julia Wang
• Created: 02/Feb/17 02:07
• Updated: 04/Aug/18 01:14

Description

IMRU Metrics is to provide metrics data to the system so that it can be shown to the user for monitoring or diagnosis. The goal is to build an E2E flow with simple/basic metrics data. We can then add more data later.

• IMetricsProvider - there are multiple sources of metrics data:
  1.Task metrics. This is in particular for IMRU task such as current iteration, progress. Each task can send task state back to driver and let driver to aggregate it. Alternatively, as UpdateTask knows current iterations and progress, to start with, we can just get task status from update task. The task metrics can be provided by task function like IUpdateFunction and send to driver by task host as TaskMessage with heartbeat.
  2. Driver metrics – For IMRU driver, it can be system state such as WaitingForEvaluator or TasksRunning, current retry number, etc. Those driver states are maintained inside IMRU driver.
  3. IMRUDriver will implement IMetricsProvider and supply metrics data.
• IMetricsSink – the metrics data will be output somewhere so that it can be consumed by a monitoring tool. An interface IMetricsSink will be defined to sink metrics data. An implementation of the interface can store the data to a remote storage. Multiple sinks can be injected.

• MetricsManager – It schedules a timer to get metrics from IMetricsProviders and output the metrics data with IMetricsSinks

Attached file shows the diagram of the design.

Issue Links

• contains: REEF-1748 Aggregate counters in MetricesService
• contains: REEF-1736 Set IMRU metrics data to EvaluatorMetrics
• contains: REEF-1737 Build MetricService for Metrics system
• contains: REEF-1734 Sending Evaluator Metrics as ContextMessage
• contains: REEF-1735 Define IMetricsSink interface
• contains: REEF-1738 Create Evaluator Metrics interface and implementations
• contains: REEF-1733 Define Driver Metrics
• relates to: REEF-1543 Build metrics sytem on top of APIs in REEF-1481
• links to: GitHub Pull Request #1460