[FLINK-25792] Async Sink Base is too being flushed too frequently resulting in backpressure even when buffer is near empty

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Zichen Liu
• Reporter: Zichen Liu
• Created: 24/Jan/22 19:06
• Updated: 01/Mar/22 11:22
• Resolved: 01/Mar/22 11:22

Description

Bug:

Async Sink Base is too being flushed too frequently resulting in backpressure even when buffer is near empty

Cause:

During a write(), flushIfAble() is called, which checks if the number of buffered elements is greater than a batch size, and if so, insists that the sink flushes immediately, even if the number of inFlightRequests is greater than the maximum allowed number of inFlightRequests, resulting in a yield of the current mailbox thread, and hence blocks.

Notice that this can occur even if the buffer is near empty, so the blocking behaviour is unnecessary and undesirable, since we would like the element to be written to the buffer and no blocking to occur.

Issue Links

• is caused by: FLINK-24041 [FLIP-171] Generic AsyncSinkBase
• is caused by: FLINK-24905 [FLIP-171] KDS implementation of Async Sink Table API
• is cloned by: FLINK-25976 Update the KDS and KDF Sink's defaults & update the docs
• is related to: FLINK-25793 Fix high throughput rate of AsyncSinkWriter for throttled destination
• links to: GitHub Pull Request #18651