[FLINK-28934] Pulsar Source put all the splits to only one parallelism when using Exclusive subscription

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.14.5, 1.15.1, 1.16.0
• Fix Version/s: 1.16.0, 1.14.6, 1.15.3
• Component/s: Connectors / Pulsar
• Labels: bug pull-request-available stale-critical test-stability
• Assignee: Yufan Sheng
• Reporter: Yufan Sheng
• Created: 11/Aug/22 15:29
• Updated: 16/Sep/22 01:38
• Resolved: 16/Sep/22 01:38

Description

The image here shows if we start a Flink application with four parallelism and four splits. All the splits would be sent to the first added reader. This is because we don't assign splits by pre-divide splits according to the size of parallelism. The readers are added to the enumerator one by one in the first bootstrap.

Issue Links

• fixes: FLINK-27388 PulsarSourceEnumeratorTest#discoverPartitionsTriggersAssignments seems flaky
• fixes: FLINK-27400 Pulsar connector subscribed the system topic when using the regex
• fixes: FLINK-27611 ConcurrentModificationException during Flink-Pulsar checkpoint notification
• fixes: FLINK-28084 Pulsar unordered reader should disable retry and delete reconsume logic.
• links to: GitHub Pull Request #20725