[HUDI-3180] Include only files belonging to completed commits while bootstrapping metadata table

• Type: Task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: writer-core
• Labels: pull-request-available sev:critical
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 05/Jan/22 22:16
• Updated: 08/Feb/22 04:31
• Resolved: 08/Feb/22 04:31

Description

metadata table bootstrap does not filter for completed commit files, but all files based on fs.list call. We should filter out files that are part of an ongoing commit while doing bootstrap. 

Issue Links

• is a parent of: HUDI-3343 Metadata Table includes Uncommitted Log Files during Bootstrap
• links to: GitHub Pull Request #4519