[HUDI-7030] Log reader data lost as that not consistent behavior in timeline's containsInstant

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 1.0.0-beta1, 0.14.1
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: ann
• Created: 03/Nov/23 12:21
• Updated: 08/Nov/23 06:40
• Resolved: 08/Nov/23 06:40

Description

Log reader filtered all log data blocks which come from inflight instant.

containsInstant return false when input instant's timestamp is not equal as anyone instant timestamp in inflight timeline.

But now, in timeline's containsInstant that input is instant's timestamp, it would return true.

 

When input is the instant with default_millis_ext, instant's timestamp is less than someone instant timestamp in timeline.

In finally, log reader skipped the completed delta commit instant and caused data lost.

I think timeline's containsInstant should have consistent behavior and update containsOrBeforeTimelineStarts to containsInstant

Issue Links

• links to: GitHub Pull Request #9982