[HUDI-8654] Support correct merging results with record positions in log blocks generated during pending compaction - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Patch Available
Priority: Blocker
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: 1.0.1
Component/s: None
Labels:
- pull-request-available

Description

When there is a pending compaction, the new base files to be generated by compaction is not available during this transaction. Given the log files in MOR from this transaction can be attached to the base file generated by the compaction in the latest file slice, the accurate record positions may not be derived. However, the log files written in later delta commits after completed compaction have accurate positions.

Similarly, for NBCC, the compaction can be schedule during an inflight deltacommit, and in this case the log file generated by the inflight deltacommit is associated with the new base file from the compaction, which may have different positions because of deletes.

We need to make sure that the file group reader with position-based merging generates the correct results in such mix of log blocks.