[HUDI-2926] Inline compaction of MOR table does not merge last log blocks when metadata table is enabled

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Assignee: Prashant Wason Assign to me
• Reporter: Prashant Wason
• Created: 03/Dec/21 08:14
• Updated: 13/Jan/22 22:25
• Resolved: 13/Jan/22 22:25

Description

This issue happens when:

1. Metadata Table is enabled
2. It is a MOR Table
3. Append is not supported

See HUDI-2925 for more detailed reasoning in the context of clean operations.

 

Summary: HoodieTable has a cached metadata reader which should be synced to pick up changes on the dataset.

Assume last deltacommit was 99.deltacommit and the log file version is 15. The next commit will be 100.deltacommit which will write log file 16 (new log file as APPEND not supported). If postWrite we read from the cached metadata reader  then it will not report log version 16.