[FLINK-17865] Increase default size of `state.backend.fs.memory-threshold`

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Done
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available

Release Note

After FLINK-17865 the default value of `state.backend.fs.memory-threshold` has been increased from 1K to 20K to prevent too many small files created on remote FS for small states. NOTICE: jobs with large parallelism on source or stateful operators may have "JM OOM" or "RPC message exceeding maximum frame size" problem with this change, and please manually set the configuration back to 1K if you encounter such issues.

• Assignee: Yun Tang
• Reporter: Yu Li
• Created: 21/May/20 11:37
• Updated: 29/May/20 01:21
• Resolved: 28/May/20 18:02

Description

As discussed in ML thread, we decided to increase the default value of `state.backend.fs.memory-threshold` from 1K to 20K to prevent too many small files created on remote FS.

We need to add an explicit release note to notify our users about this change, especially for those having jobs with large parallelism on source or stateful operators.

Issue Links

• links to: GitHub Pull Request #12282
• links to: GitHub Pull Request #12373