[FLINK-3084] File State Backend should not write very small state into files

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 1.0.0
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 26/Nov/15 16:23
• Updated: 28/Feb/19 11:14
• Resolved: 01/Dec/15 08:45

Description

Currently, the FsStateBackend writes all state into files. Some state (like Kafka Offsets) is so small that it adds unnecessary overhead, and sometimes the checkpointed file handles are larger than the actual state.

Small state (below a certain threshold, say 1 KB) should not be stored in files, but directly in the state handles.