[BEAM-2847] Stop streaming pipeline when global watermark reaches +INF

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-spark
• Assignee: Unassigned
• Reporter: Aviem Zur
• Created: 06/Sep/17 13:35
• Updated: 03/Jun/22 18:33

Description

Stop streaming pipeline when global watermark reaches +INF as suggested in BEAM-849 comments.

We can do this similarly to the solution in TestSparkRunner#awaitWatermarksOrTimeout by polling

SparkTimerInternals.global(GlobalWatermarkHolder.get(...))

We can also remove this code from TestSparkRunner if we make sure all sources in test behave in a way which shifts the watermark to +INF at the end of the test.

Issue Links

• is related to: BEAM-160 Port 'NexMark Queries' to Beam for use as integration test