[FLINK-35555] Serializing List with null values throws NPE

• Type: Sub-task
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.20.0
• Fix Version/s: 1.20.0
• Component/s: API / Type Serialization System
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Zhanghao Chen
• Created: 07/Jun/24 11:39
• Updated: 08/Jun/24 03:33

Description

FLINK-34123 introduced built-in serialization support for java.util.List, which relies on the existing ListSerializer impl. However, ListSerializer does not allow null values, as it is originally designed for serializing ListState only where null value is explicitly forbidden in the contract.

Directly adding null marker to allow null values will break backwards state compatibility, so we'll need to introduce a new List serializer called NullableElementListSerializer and the corrsponding TypeInformation called NullableElementListTypeInfo that allows null values for serializing user objects, and leaves the existing ListSerializer and ListTypeInfo for Flink's internal state use.

FLINK-23420 is similar to our case here. We can extend ListSerializer to allow null values via  a binary mask for marking null values and rely on TypeSerializerSnapshot to deal with state-compatibility.

Issue Links

• is caused by: FLINK-34123 Introduce built-in serialization support for Map and List
• links to: GitHub Pull Request #24916