[HUDI-3343] Metadata Table includes Uncommitted Log Files during Bootstrap

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: metadata
• Story Points: 0
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin

Reviewers

sivabalan narayanan, Y Ethan Guo

• Created: 28/Jan/22 23:16
• Updated: 21/Feb/22 14:18
• Resolved: 02/Feb/22 21:29

Description

While working on a fix for HUDI-3322, discovered a following issue:

If we're bootstrapping the MT during pending Rollback operation (this could happen when previous writer had MT disabled when writing the data), since bootstrapping is done after Rollback is executed (with its side-effects already being reflected on FS) bootstrapping would incorrectly include intermediary files created by the Rollback (like log-files being created with Rollback Command Block appended).

 

Filtering of the files is performed here: 

https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java#L752

 

Issue Links

• Discovered while testing: HUDI-3322 Rollback Plan for Delta Commits constructed incorrectly
• is a child of: HUDI-3180 Include only files belonging to completed commits while bootstrapping metadata table
• relates to: HUDI-3462 List of fixes to Metadata table after 0.10.1
• links to: Github PR 4716