[HUDI-1127] Handling late arriving Deletes

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 0.9.0
• Component/s: deltastreamer, writer-core
• Labels: sev:high
• Story Points: 4
• Epic Link: Streaming Read
• Assignee: Unassigned
• Reporter: Bhavani Sudha
• Created: 27/Jul/20 19:31
• Updated: 15/Mar/22 03:38
• Resolved: 15/Mar/22 03:38

Description

Recently I was working on a PR to enhance OverwriteWithLatestAvroPayload class to consider records in storage when merging. Briefly, this class will ignore older updates if the record in storage is the latest one ( based on the Precombine field). 

Based on this, the expectation is that we handle any write operation that should be dealt with the same way - if they are older they should be ignored. While at this, I identified that we cannot handle all Deletes the same way. This is because we process deletes in two ways mainly -

• by adding and enabling a metadata field  `_hoodie_is_deleted` to our in the original record and sending it as an UPSERT operation.
• by using an empty payload using the EmptyHoodieRecordPayload and sending the write as a DELETE operation. 

While the former has ordering field and can be processed as expected (older deletes will be ignored), the later does not have any ordering field to identify if its an older delete or not and hence will let the older delete to go through.

Just opening this issue to track this gap. We would need to identify what is the right choice here and fix as needed.

Issue Links

• is duplicated by: HUDI-2752 The MOR DELETE block breaks the event time sequence of CDC