[BEAM-10676] Timers use the input timestamp as the timer output timestamp which prevents watermark progress

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.24.0
• Component/s: sdk-py-core, sdk-py-harness
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 11/Aug/20 15:44
• Updated: 4 days ago 07:50
• Resolved: 15/May/21 17:56

Description

By default, the Python SDK adds a timer output timestamp equal to the current timestamp of an element. This is problematic because

1. We hold back the output watermark on the current element's timestamp for every timer
2. It doesn't match the behavior in the Java SDK which defaults to using the fire timestamp as the timer output timestamp (and adds a hold on it)
3. There is no way for the user to influence this behavior because there is no user-facing API

https://github.com/apache/beam/blob/dfadde2d3ee0a0487362dbcca80388fdc2ef2302/sdks/python/apache_beam/runners/worker/bundle_processor.py#L650

We should use the fire timestamp as the default output timestamp.

Issue Links

• relates to: BEAM-10691 FlinkRunner: pipeline slows down due to expensive output timestamp queue
• links to: GitHub Pull Request #12531