[FLINK-27255] Flink-avro does not support serialization and deserialization of avro schema longer than 65535 characters

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.4
• Fix Version/s: 1.14.5, 1.15.1, 1.16.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available

Language

Java

• Assignee: Haizhou Zhao
• Reporter: Haizhou Zhao
• Created: 15/Apr/22 00:15
• Updated: 11/Apr/23 13:19
• Resolved: 10/May/22 23:25

Description

The underlying serialization of avro schema uses string serialization method of ObjectOutputStream.class, however, the default string serialization by ObjectOutputStream.class does not support handling string of more than 66535 characters (64kb). As a result, constructing flink operators that input/output Avro Generic Record with huge schema is not possible.

 

The purposed fix is two change the serialization and deserialization method of these following classes so that huge string could also be handled.

 

GenericRecordAvroTypeInfo

SerializableAvroSchema

 

Issue Links

• is duplicated by: FLINK-22727 SerializableAvroSchema cannot handle large schema
• links to: GitHub Pull Request #19645
• links to: GitHub Pull Request #19704
• links to: GitHub Pull Request #19705
• links to: GitHub Pull Request #19719