[FLINK-20213] Partition commit is delayed when records keep coming

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.2
• Fix Version/s: 1.11.3, 1.12.1
• Component/s: Connectors / FileSystem, Table SQL / Ecosystem
• Labels: pull-request-available
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 18/Nov/20 09:46
• Updated: 28/Jan/21 08:21
• Resolved: 22/Dec/20 02:15

Description

When set partition-commit.delay=0, Users expect partitions to be committed immediately.

However, if the record of this partition continues to flow in, the bucket for the partition will be activated, and no inactive bucket will appear.

We need to consider listening to bucket created.

Issue Links

• fixes: FLINK-20771 Hive partition is not added when there is a lot of data
• relates to: FLINK-20671 Partition doesn't commit until the end of partition
• links to: GitHub Pull Request #14116
• links to: GitHub Pull Request #14442