[FLINK-20217] More fine-grained timer processing

• Type: Improvement
• Status: Closed
• Priority: Not a Priority
• Resolution: Fixed
• Affects Version/s: 1.10.2, 1.11.2, 1.12.0
• Fix Version/s: 1.20.0
• Component/s: API / DataStream, Runtime / Task
• Labels: auto-deprioritized-major auto-deprioritized-minor pull-request-available

Release Note

Firing timers can now be interrupted to speed up checkpointing. Timers that were interrupted by a checkpoint, will be fired shortly after checkpoint completes. By default this features is disabled. To enabled it please set:

execution.checkpointing.unaligned.interruptible-timers.enabled

to true. Currently supported only by all TableStreamOperators and CepOperator.

• Assignee: Piotr Nowojski
• Reporter: Nico Kruber
• Created: 18/Nov/20 11:52
• Updated: 13/Jun/24 09:16
• Resolved: 12/Jun/24 12:30

Description

Timers are currently processed in one big block under the checkpoint lock (under InternalTimerServiceImpl#advanceWatermark. This can be problematic in a number of scenarios while doing checkpointing which would lead to checkpoints timing out (and even unaligned checkpoints would not help).

If you have a huge number of timers to process when advancing the watermark and the task is also back-pressured, the situation may actually be worse since you would block on the checkpoint lock and also wait for buffers/credits from the receiver.

I propose to make this loop more fine-grained so that it is interruptible by checkpoints, but maybe there is also some other way to improve here.

This issue has been for example observed here: https://lists.apache.org/thread/f6ffk9912fg5j1rfkxbzrh0qmp4w6qry

Issue Links

• is related to: FLINK-31370 Cancellation of the StreamTask should prevent more timers from being fired
• links to: GitHub Pull Request #24895
• mentioned in: Page Loading...

Sub-Tasks

• Skip execution of interruptible mails when yielding: Closed