[FLINK-25402] FLIP-198: Working directory for Flink processes

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 21/Dec/21 15:05
• Updated: 06/Feb/23 05:23
• Resolved: 19/Jan/22 15:30

Description

This issue implements FLIP-198. The idea is to introduce a working directory that can be used by Flink processes to store data that can be recovered if the process is restarted with access to the same working directory (e.g. via persistent volumes or restarting the process on the same node).

The working directory will be default created in the temporary directory of the executing node. Moreover, the path will include the resource id of the Flink process. This will ensure that by default (random resource id), a newly started Flink process will see an empty working directory.

This is an optional feature that might not be usable for all Flink deployments.

Issue Links

• causes: FLINK-27165 Benchmark test SerializationFrameworkMiniBenchmarks#serializerHeavyString became unstable
• is required by: FLINK-25403 Make BlobServer/BlobCache compatible with local working directories
• relates to: FLINK-26014 Document how to use the working directory for faster local recoveries
• links to: FLIP-198
• links to: Github PR #18083
• links to: GitHub Pull Request #18083