[FLINK-25962] Flink generated Avro schemas can't be parsed using Python

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.3
• Fix Version/s: 1.16.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available
• Release Note: Avro schemas generated by Flink now use the "org.apache.flink.avro.generated" namespace for compatibility with the Avro Python SDK.
• Assignee: Ryan Skraba
• Reporter: Ryan Skraba
• Created: 04/Feb/22 17:11
• Updated: 02/Aug/22 12:56
• Resolved: 02/Aug/22 12:56

Description

Flink currently generates Avro schemas as records with the top-level name "record"

Unfortunately, there is some inconsistency between Avro implementations in different languages that may prevent this record from being read, notably Python, which generates the error:
avro.schema.SchemaParseException: record is a reserved type name
(See the comment on FLINK-18096 for the full stack trace).

The Java SDK accepts this name, and there's an ongoing discussion about what the expected behaviour should be.  This should be clarified and fixed in Avro, of course.

Regardless of the resolution, the best practice (which is used almost everywhere else in the Flink codebase) is to explicitly specify a top-level namespace for an Avro record.   We should use a default like: org.apache.flink.avro.generated.

Issue Links

• is related to: FLINK-18096 Generated avro formats should support user specified name and namespace
• links to: GitHub Pull Request #18635