[HUDI-2303] TestMereIntoLogOnlyTable with metadata enabled surfaces likely bug

• Type: Task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Common Core
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Prashant Wason
• Reporter: Udit Mehrotra
• Created: 12/Aug/21 22:09
• Updated: 04/Jan/22 00:09
• Resolved: 03/Nov/21 12:29

Description

While enabling Metadata as part of https://github.com/apache/hudi/pull/3411/ one of the test that fails is TestMereIntoLogOnlyTable.

Upon looking a bit, what I found is after the final Merge command there is an inline compaction that is triggered. The parquet file formed as part of the compaction misses out on the data from the latest log file right before compaction.

I think it might be because of metadata returning an incorrect list for compaction, missing out on the latest log file.

cc pwason vinoth

 

 

Issue Links

• is a child of: HUDI-2276 Enable Metadata Table by default for both writers and readers