[FLINK-26726] Remove the unregistered task from readersAwaitingSplit

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0, 1.17.0, 1.15.3, 1.14.7
• Component/s: Connectors / Hive, Table SQL / Ecosystem
• Labels: pull-request-available stale-assigned
• Assignee: zoucao
• Reporter: zoucao
• Created: 18/Mar/22 08:48
• Updated: 14/Oct/22 02:28
• Resolved: 14/Oct/22 02:28

Description

Recently, we faced a problem caused by the unregistered task when using the hive table as a source to do streaming reading.
I think the problem is that we do not remove the unregistered task from `readersAwaitingSplit` in `ContinuousHiveSplitEnumerator` and `ContinuousFileSplitEnumerator`.

Assuming that we have two tasks 0 and 1, they all exist in `readersAwaitingSplit`, if there does not exist any new file in the path for a long time. Then, a new split is generated, and it is assigned to task-1. Unfortunately, task-1 can not consume the split successfully, and the exception will be thrown and cause all tasks to restart. The failover will not affect the `readersAwaitingSplit`, but it will clear the `SourceCoordinatorContext#registeredReaders`.
After restarting, task-0 exists in `readersAwaitingSplit` but not in `registeredReaders`. if task-1 register first and send the request to get split, the SplitEnumerator will assign splits for both task-1 and task-0, but task-0 has not been registered.

The stack exists in the attachment.

Issue Links

• links to: GitHub Pull Request #19250
• links to: GitHub Pull Request #21036
• links to: GitHub Pull Request #21038
• links to: GitHub Pull Request #21044