Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Related issue: https://issues.apache.org/jira/browse/HUDI-5553
The specific problem is that when concurrent replace commit operations are executed, two replace commits may point to the same file ID, resulting in a duplicate key error. The existing issue solved the problem of scheduling delete partition while there are pending clustering or compaction operations, which will be prevented in this case. However, this solution is not perfect and may still cause data inconsistency if a clustering plan is scheduled before the delete partition is committed. Because validation is one-way.In this case, both replace commits will still contain duplicate keys, and the table will become inconsistent when both plans are committed. This is very fatal, and there are other similar scenarios that may bypass the validation of the existing issue. Moreover, the existing issue is at the partition level and is not precise enough.
Here is my solution:
As shown in the figure, both drop partition and clustering will go through a period of time that is not registered to the timeline, which is the scenario that the previous issue did not solve. Here, I register the replace file IDs involved in each replace commit to the active timeline (the replace commit timeline that has been submitted has saved partitionToReplaceFileIds, and only pending requests need to be processed). Since in the case of Spark SQL, delete partition creates a requested commit in advance during write, which is inconvenient to handle, I save the pending replace commit's partitionToReplaceFileIds information to the inflight commit's extra metadata. Therefore, each time drop partition or clustering is executed, it only needs to read the partitionToReplaceFileIds information in the timeline after ensuring that the inflight commit information has been saved to the timeline to ensure that there are no duplicate file IDs and prevent this kind of error from occurring.
In simple terms, each replace commit will register the replace file ID information to the timeline whether it is submitted or not, at the same time, each submission will check this information to ensure that it will not be repeated, so that any replace commit containing this file ID will be prevented, ensuring that there are no duplicate keys.
When this idea is also implemented on the compaction commit, the modification involved in the related issue can be removed.
Attachments
Issue Links
- links to