[FLINK-28870] Pulsar Source hangs on the small incoming message rates

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: pulsar-4.0.0
• Component/s: Connectors / Pulsar
• Labels: pull-request-available
• Assignee: Yufan Sheng
• Reporter: xie
• Created: 08/Aug/22 10:57
• Updated: 10/Jan/23 12:20
• Resolved: 10/Jan/23 08:40

Description

When using Pulsar Source to consume data, if the data rate is small, e.g. 2 msg/s, there will be long periods of time when no messages are consumed. This is caused by the default PulsarSourceOptions.PULSAR_MAX_FETCH_TIME and PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS. Pulsar Source will try to pull messages until any one of the condition exceed.

 

Issue Links

• links to: GitHub Pull Request #15