[BEAM-2456] Introduce generic metric poll thread interceptor & generic sink

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-ideas
• Assignee: Jean-Baptiste Onofré
• Reporter: Jean-Baptiste Onofré
• Created: 16/Jun/17 07:05
• Updated: 16/May/20 13:51
• Resolved: 09/Jan/19 12:40

Description

The Spark runner provides a convenient feature which is the metric sink.

By configuration, it allows us to configure a metric sink using metrics.properties configuration containing:

driver.sink.graphite.class=org.apache.beam.runners.spark.metrics.sink.GraphiteSink
driver.sink.graphite.host=localhost
driver.sink.graphite.port=2003
driver.sink.graphite.prefix=spark
driver.sink.graphite.period=1
driver.sink.graphite.unit=SECONDS 

This approach is very convenient to send the metric to any sink. I think we can apply this logic in generic way working with any runner.

The idea is to use a metric poll thread in the pipeline (that we can enable via PipelineOptions) and send to a sink.

I started a PoC about that.

Issue Links

• is superceded by: BEAM-3310 Push metrics to a backend in an runner agnostic way