[HUDI-5176] Incremental source may miss commits if there are inflight commits before completed commits

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.12.2
• Component/s: incremental-query
• Labels: pull-request-available
• Story Points: 2
• Assignee: Ethan Guo
• Reporter: Ethan Guo
• Created: 08/Nov/22 06:22
• Updated: 02/Jun/23 16:10
• Resolved: 15/Nov/22 02:36

Description

Consider the following scenario of concurrent writers. Writer 1 starts a commit at t1 and later writer 2 starts another commit at t2 (t2 > t1). Commit t2 finishes earlier than t1.

---------------------------------------------------------> t
 instant t1 |------------------------------| (writer 1)
 instant t2         |--------------|         (writer 2) 

This leaves an inflight commit (t1) before a completed commit (t2) on the Hudi timeline.  Given that the incremental pull uses only completed commits to determine the start and end instants for incremental query and advance the checkpoint, the data for the inflight commits may never be pulled from the incremental source.

 

Issue Links

• is related to: HUDI-6312 Guard incremental query read paths for multi-writer scenario
• links to: GitHub Pull Request #7160