[HUDI-2119] Syncing of rollbacks to metadata table does not work in all cases

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Labels: pull-request-available release-blocker
• Assignee: Ethan Guo
• Reporter: Prashant Wason
• Created: 01/Jul/21 20:08
• Updated: 01/Oct/21 15:52
• Resolved: 15/Aug/21 02:51

Description

This is an issue with inline automatic rollbacks.

Metadata table assumes that a rollbacks is to be applied if the instant-being-rolled back has a timestamp less than the last deltacommit time on the metadata timeline. We do not explicitly check if the instant-being-rolled-back was actually written to metadata table.

**A rollback adds a record to metadata table which "deletes" files from a failed/earlier commit. If the files being deleted were never actually committed to metadata table earlier, the deletes cannot be consolidated during metadata table reads. This leads to a HoodieMetadataException as we cannot differentiate this from a bug where we might have missed committing a commit to metadata table.

 

 

Issue Links

• links to: GitHub Pull Request #3210