[BEAM-5807] Add Schema support for AVRO

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.10.0
• Component/s: dsl-sql
• Assignee: Gleb Kanterov
• Reporter: Gleb Kanterov
• Created: 22/Oct/18 09:06
• Updated: 16/May/20 13:34
• Resolved: 14/Feb/19 23:37

Description

1. Motivation

At the moment the only way to read AVRO is through code generation with avro-compiler and JavaBeanSchema. It makes it not possible to write transforms that can work with dynamic schemas. AVRO has generic data type called GenericRecord, reading is implemented in AvroIO.

readGenericRecords. There is a code to convert GenericRecord to Row shipped as a part of BigQueryIO. However, it doesn't support all types and nested records.

Issue Links

• links to: GitHub Pull Request #6777