[HUDI-6888] Optimize scanInternalv2 to single pass

• Type: Task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: table-service
• Labels: pull-request-available
• Epic Link: RFC-48 Minor compaction
• Assignee: Sagar Sumit
• Reporter: Sagar Sumit
• Created: 21/Sep/23 16:25
• Updated: 24/Jul/24 07:14

Description

The current algorithm take two passes over the log blocks: # First pass to collect all the valid blocks alongwith block instant times including rollback block's target instant time.

1. Second pass, in rever order of block instant time, to track final compacted instant times for each block.

Now that we have removed appending to the same log file for multiple deltacommits, we can probably scan in single pass by keeping an active list or hash map of block times to their corresponding block, updating as we go. Should be tested for:

1. Out of order merged blocks: Log compaction is scheduled and by the time it appended a block, another block is added by another writer.
2. Log compaction operation failed, so a rollback is issued for this block. Here the rollback can be next block or can come at a later point of time.
3. Log compaction is executing and, before committing, compaction starts running on the same file group.

Issue Links

• links to: GitHub Pull Request #9762