[BEAM-7829] AvroUtils.toAvroSchema should put a Schema name to pass Avro Schema validation

• Type: Test
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.16.0
• Component/s: io-java-avro, sdk-java-core
• Labels: Done
• Assignee: Ryan Skraba
• Reporter: Ismaël Mejía
• Created: 26/Jul/19 08:59
• Updated: 06/Oct/20 21:59
• Resolved: 29/Aug/19 12:45

Description

While trying to use an Avro PCollection with the SQL transform I notice you could not do correctly a bijective transform: PCollection<GenericRecord> -> SQL -> PCollection<Row> -> ParDo -> PCollection<GenericRecord> I noticed that some of the Avro metadata gets lost in particular the name of the Avro Schema. This is important because Avro validates that the schema has a name and if it does not it breaks with a ParseException.

org.apache.avro.SchemaParseException: Illegal character in: EXPR$1
    at org.apache.avro.Schema.validateName (Schema.java:1151)
    at org.apache.avro.Schema.access$200 (Schema.java:81)
    at org.apache.avro.Schema$Field.<init> (Schema.java:403)
    at org.apache.avro.Schema$Field.<init> (Schema.java:423)
    at org.apache.avro.Schema$Field.<init> (Schema.java:415)

Issue Links

• is related to: AVRO-2491 Schema.createRecord allows to create non parseable Schemas
• links to: GitHub Pull Request #9247