Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.15.0
Description
In the current implementation of FsStateChangelogStorage, dstl files from all jobs are put into the same directory (configured via dstl.dfs.base-path). Everything is fine if it's a filesystem like S3.But if it is a file system like hadoop, there will be some problems.
First, there may be an upper limit to the number of files in a single directory. Increasing this threshold will greatly reduce the performance of the distributed file system.
Second, dstl file management becomes difficult because the user cannot tell which job the dstl file belongs to, especially when the retained checkpoint is turned on.
Propose
- create a subdirectory named with the job id under the dstl.dfs.base-path directory when the job starts
- all dstl files upload to the subdirectory
( Going a step further, we can even create two levels of subdirectories under the dstl.dfs.base-path directory, like base-path/{jobId}/dstl . This way, if the user configures the same dstl.dfs.base-path as state.checkpoints.dir, all files needed for job recovery will be in the same directory and well organized. )
Attachments
Issue Links
- links to