[HUDI-4083] Fix the flink application fails to start due to uncompleted archiving deletion

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Unresolved
• Component/s: archiving
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Ying Lin
• Created: 11/May/22 07:28
• Updated: 26/May/22 03:41

Description

Suppose a flink application crashes while archiving, it leaves some instant files that should be deleted.

If the commit file is deleted but the in-flight file is left, when the flink application starts, it will scan and get the last pending instant and throw an exception.

So we need to delete non-commit file first, and then delete commit file.

Issue Links

• links to: GitHub Pull Request #5556