[HUDI-7687] Instant should not be archived until replaced file groups or older file versions are deleted

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: archive clean
• Assignee: sivabalan narayanan
• Reporter: Krishen Bhan
• Created: 29/Apr/24 22:51
• Updated: 29/Apr/24 23:02

Description

When archival runs it may consider an instant as a candidate for archival even if the file groups said instant replaced/updated still need to undergo a `clean`. For example, consider the following scenario with clean and archived scheduled/executed independently in different jobs

1. Insert at C1 creates file group f1 in partition
2. Replacecommit at RC2 creates file group f2 in partition, and replaces f1
3. Any reader of partition that calls HUDI API (with or without using MDT) will recognize that f1 should be ignored, as it has been replaced. This is since RC2 instant file is in active timeline
4. Some more instants are added to timeline. RC2 is now eligible to be cleaned (as per the table writers' clean policy). Assume though that file groups replaces by RC2 haven't been deleted yet, such as due to clean repeatedly failing, async clean not being scheduled yet, or the clean failing to delete said file groups.
5. An archive job eventually is triggered, and archives C1 and RC2. Note that f1 is still in partition

Now the table has the same consistency issue as seen in https://issues.apache.org/jira/browse/HUDI-7655 , where replaced file groups are still in partition and readers may see inconsistent data. 

 

This situation can be avoided by ensuring that archival will "block" and no go past an older instant time if it sees that said instant didn't undergo a clean yet.