[BEAM-2186] Use Accumulable MetricsContainers in the DirectRunner

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-direct
• Assignee: Unassigned
• Reporter: Ben Chambers
• Created: 05/May/17 21:14
• Updated: 03/Jun/22 17:18

Description

Once PR 2649 is submitted, we should be able to use the accumulable functionality to clean up the DirectRunner. Since this runner updates metrics from multiple processing threads simultaneously, we will need to figure out how to either (1) make the accumulable behavior thread safe or (2) have a non-blocking way of performing the accumulation sequentially.

If we go with (2), we can put updated containers in a queue and then have a single thread trying to apply those updates.

https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java#L912)