[FLINK-34228] Add long UTF serializer/deserializer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.20.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Peter Vary
• Created: 24/Jan/24 12:43
• Updated: 19/Mar/24 08:36
• Resolved: 19/Mar/24 08:36

Description

DataOutputSerializer.writeUTF has a hard limit on the length of the string (64k). This is inherited from the DataOutput.writeUTF method, where the JDK specifically defines this limit [1].

For our use-case we need to enable the possibility to serialize longer UTF strings, so we will need to define a writeLongUTF method with a similar specification than the writeUTF, but without the length limit.

Based on the discussion on the mailing list, this is a good additional serialization utility to Flink [2]

[1] - https://docs.oracle.com/javase/8/docs/api/java/io/DataOutput.html#writeUTF-java.lang.String-
[2] - https://lists.apache.org/thread/ocm6cj0h8o3wbwo7fz2l1b4odss750rk

Issue Links

• links to: GitHub Pull Request #24191