[BEAM-579] Integrate NamedAggregators into Spark's sink system

• Type: Task
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 0.3.0-incubating
• Component/s: runner-spark
• Assignee: Stas Levin
• Reporter: Stas Levin
• Created: 23/Aug/16 09:07
• Updated: 16/May/20 13:31
• Resolved: 26/Aug/16 10:18

Description

At the moment NamedAggregators is an adapter between Beam's Aggregator and Spark's Accumulator and is implemented as a single Spark Accumulator, holding a map of metrics that can be augmented with new metrics dynamically, after the the pipeline has already started.

Spark's out-of-the-box metrics mechanism does not support adding metrics to Source s that have already registered (it pulls their metrics upon registration and never updates them again).

In light of the above, it would seem that there is a gap to bridge between the dynamic nature of NamedAggregators and Spark's current metric system so that metrics that are added dynamically are also reported to the defined Spark Sink s.

Issue Links

• is related to: BEAM-628 Sink class org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink cannot be instantiated
• links to: GitHub Pull Request #867