[HUDI-6371] IndexingCatchupTask does not handle failed tasks

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Labels: indexing writer
• Story Points: 8
• Epic Link: Async Metadata Indexing
• Assignee: Sagar Sumit
• Reporter: Prashant Wason
• Created: 14/Jun/23 06:22
• Updated: 13/Oct/24 22:38

Description

IndexingCatchupTask::run() iterates over all the commits to sync them to MDT. It will sleep while the commit is inflight. But if the commit actually failed then it will forever be in inflight mode and hence the run will never progress.

We should use HB to detect failed commits and ignore them.