We have discovered the following issues while using the Metadata Table code in production:
Issue 1: Automatic rollbacks during commit get a timestamp which is out of order
Suppose commit C1 failed. The next commit will try to rollback C1 automatically. This will create the following two instances C2.commit and R3.rollback. Hence, the rollback will have a timestamp > the commit which occurs after it.
This is because of how the code is implemented in AbstractHoodieWriteClient.startCommitWithTime() where the timestamp of the next commit is chosen before the timestamp of the rollback instant.
Issue 2: Syncing of rollbacks is not working
Due to the above HUDI issue, syncing of rollbacks in Metadata Table does not work correctly.
Assume the timeline as follows:
Dataset Timeline: C1 C2. C3
Metadata Timeline: DC1 DC2. (dc=delta-commit)
Suppose the next commit C4 fails. When C5 is attempted, C4 will be automatically tolled back. Due to the issue #1, the timelines will become as follows:
Dataset Timeline: C1 C2. C3. C5 R6
Metadata Timeline: DC1 DC2
Now if the Metadata Table is synced (AbstractHoodieWriteClient.postCommit), the code will end up processing C5 first and then R6 which will mean that the file rolled back in R6 will be committed to the metadata table as deleted files. There is logic within HoodieTableMetadataUtils.processRollbackMetadata() to ignore R6 in this scenario but it does not work because of the issue #1.
Issue #3: Rollback instants are deleted inline
Current rollback code deleted older instants inline. The delete logic keeps oldest ten instants (hardcoded) and removes all more-recent rollback instants. Furthermore, the deletion ONLY deletes the rollback.complete and does not remove the corresponding rollback.inflight files.
Hence, will many rollbacks the following timeline is possible
Timeline: C1. C2 C3 C4. R5.inflight C5 C6 C7 ...
(there are 9 previous rollback instants to R5).
Issue #4: Metadata Table reader does not show correct view of the metadata
Assume the timeline is as in Issue #3 with a leftover rollback.inflight instant. Also assume that the metadata table is synced only till C4. The MetadataTableWriter will not sync any more instants to the Metadata Table since an incomplete instant is present next.
The same sync logic is also used by the MetadataReader to perform the in-memory merge of timeline. Hence, the reader will also not consider C5, C6 and C7 thereby providing an incorrect and older view of the FileSlices and FileGroups.
Any future ingestion into this table MAY insert data into older versions of the FileSlices which will end up being a data loss when queried.