Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.6.0, 3.0.0-alpha-4, 2.5.2
-
None
-
Reviewed
Description
We found a a race in the CleanerChore related to back reference links. When the HFileLinkCleaner runs for a file it can make 2 decisions depending on the file types.
- HFiles, The cleaner for HFile deletion only checks if the .links-<> directory is present with files.
- Back reference links, the cleaner checks if the forward link is still available in the data directory.
The logic and order how the cleaner checks these 2 files matters. When the back reference is checked first it can remove both the reference and the HFile from the archive, however, when it first runs for the HFile then only the back-reference is removed. In this case, the HFile is only deleted in the next iteration of the CleanerChore, and it could be very slow if the list of files are huge in case of using object store.
The goal of this task is to improve traverse of the archived HFile, reusing the list of found back reference files, and immediately apply the checks for the Back reference links.
Attachments
Issue Links
- links to