[BEAM-3926] Support MetricsPusher in Dataflow Runner

• Type: Sub-task
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-dataflow
• Assignee: Unassigned
• Reporter: Scott Wegner
• Created: 26/Mar/18 16:55
• Updated: 03/Jun/22 19:26

Description

See relevant email thread. From echauchot:
 
AFAIK Dataflow being a cloud hosted engine, the related runner is very different from the others. It just submits a job to the cloud hosted engine. So, no access to metrics container etc... from the runner. So I think that the MetricsPusher (component responsible for merging metrics and pushing them to a sink backend) must not be instanciated in DataflowRunner otherwise it would be more a client (driver) piece of code and we will lose all the interest of being close to the execution engine (among other things instrumentation of the execution of the pipelines).  I think that the MetricsPusher needs to be instanciated in the actual Dataflow engine.
 
 

Issue Links

• is related to: BEAM-3967 Create a langage agnostic definition of MetricsPusher
• links to: GitHub Pull Request #5437