Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
The storage location of completed checkpoints misses a proper representation. Because of that, there is no place that can handle the deletion of a checkpoint directory, or the dropping of a checkpoint specific table.
Current workaround for file systems is, for example, that every file disposal checks if the parent directory is now empty, and deletes it if that is the case. That is not only inefficient, but prohibitively expensive on some systems, like Amazon S3.
Properly representing the storage location for completed checkpoints allows us to add a disposal call for that location.
That CompletedCheckpointStorageLocation can also be used to capture "external pointers", metadata, and even allow us to use custom serialization and deserialization of the metadata in the future, making the abstraction more extensible by allowing users to introduce new types of state handles.