Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Abandoned
-
1.4.0, 1.4.1, 1.4.2
-
None
-
None
Description
With flink 1.4.0 cleaning of checkpoints is handled on the job manager, the SharedStateRegistry uses the job manager's io executor as the executor to schedule disposals of checkpoint state on. This executor has a fixed size equal to the number of cpu cores. When a lot of small files are created by many jobs and a slow filesystem is used this may not be enough.
It would be good to make this setting configurable. Initializing an executor service with the number of cpu's on the system makes sense for cpu intensive tasks less so for io based workloads