[FLINK-5138] Add the method in state backend to estimate memory by hint of state size in ResourceSpec

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Workaround
• Component/s: Runtime / State Backends
• Assignee: Zhijiang
• Reporter: Zhijiang
• Created: 23/Nov/16 07:15
• Updated: 04/Dec/18 06:43
• Resolved: 04/Dec/18 06:43

Description

Users may specify the state size in setResources API, then the state backend would roughly estimate the different kinds of memory usages based on the state size. Currently the specific state backend (FsStateBackend, MemoryStateBackend, etc) can implement the default unknown resource for it to work around.

There are two advantages to do this:

• This estimate memory will be considered together when requesting resource for container, so the total memory usage will not exceed the container limit.
• Especially for RocksDB backend, the proper memory setting for RocksDB may get better performance in read and write operation.