[BEAM-6333] allow disabling/configuring automatic watermark generation for idle kafka partitions

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Won't Fix
• Affects Version/s: 2.9.0
• Fix Version/s: Not applicable
• Component/s: io-java-kafka
• Labels: easyfix pull-request-available
• Assignee: Raghu Angadi
• Reporter: Jan Doms
• Created: 02/Jan/19 14:08
• Updated: 16/May/20 13:55
• Resolved: 04/Jan/19 00:45

Description

For pipelines that require the emitted watermarks to be absolutely correct the current behavior using 2 seconds timeout could lead to problems.

Therefor it should be possible to disable this behavior. While changing the code the hardcoded 2 seconds can also be made configurable. (There's already a TODO about that in the code.)

 

A (preliminary) PR making this change is available: https://github.com/apache/beam/pull/7382.

Issue Links

• links to: GitHub Pull Request #7382