[FLINK-17928] Incorrect state size reported when using unaligned checkpoints

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Component/s: Runtime / Checkpointing
• Labels: pull-request-available
• Assignee: Roman Khachatryan
• Reporter: Piotr Nowojski
• Created: 25/May/20 13:29
• Updated: 28/May/20 06:15
• Resolved: 28/May/20 06:15

Description

Even when checkpoints on HDFS are between 100-300MBs, the reported state size is in orders of magnitude larger with values like:

1GiB  1.5TiB  2.0TiB  2.1TiB  2.1TiB
148GiB  148GiB  148GiB  148GiB  148GiB  148GiB

it's probably because we have multiple Collection<InputChannelStateHandle>, and each of the individual handle returns the same value from AbstractChannelStateHandle#getStateSize - the full size of the spilled data, ignoring that only small portion of those data belong to a single input channel/result subpartition. In other words {{
org.apache.flink.runtime.state.AbstractChannelStateHandle#getStateSize}} should be taking the offsets into account and return only the size of the data that belong exclusively to this handle.

Issue Links

• links to: GitHub Pull Request #12362