Details
-
Test
-
Status: Resolved
-
P3
-
Resolution: Fixed
-
None
Description
While trying to use an Avro PCollection with the SQL transform I notice you could not do correctly a bijective transform: PCollection<GenericRecord> -> SQL -> PCollection<Row> -> ParDo -> PCollection<GenericRecord> I noticed that some of the Avro metadata gets lost in particular the name of the Avro Schema. This is important because Avro validates that the schema has a name and if it does not it breaks with a ParseException.
org.apache.avro.SchemaParseException: Illegal character in: EXPR$1
at org.apache.avro.Schema.validateName (Schema.java:1151)
at org.apache.avro.Schema.access$200 (Schema.java:81)
at org.apache.avro.Schema$Field.<init> (Schema.java:403)
at org.apache.avro.Schema$Field.<init> (Schema.java:423)
at org.apache.avro.Schema$Field.<init> (Schema.java:415)
Attachments
Issue Links
- is related to
-
AVRO-2491 Schema.createRecord allows to create non parseable Schemas
-
- Resolved
-
- links to