[AVRO-3182] Avro in Java cannot parse the union schema from C#

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.10.2
• Fix Version/s: 1.11.0
• Component/s: csharp, java
• Labels: pull-request-available
• Assignee: Shawn Wu
• Reporter: Shawn Wu
• Created: 04/Aug/21 07:28
• Updated: 06/Sep/21 13:23
• Resolved: 06/Sep/21 13:23

Description

Version: 1.10.2

 

I need to transfer data from C# to Java.

When I use Avro in C#, I found Schema.ToString() in C# for union schema will add an extra type property in JSON string.

However, the correct union schema is a JSON array.

This is my test code in C#:

var columnCountSchema = Schema.Parse("{\"type\":\"record\",\"name\":\"columnCount\",\"namespace\":\"avro.test.header\",\"fields\":[{\"name\":\"columnCount\",\"type\":\"int\"}]}");
var errorMessageSchema = Schema.Parse("{\"type\":\"record\",\"name\":\"errorMessage\",\"namespace\":\"avro.test.header\",\"fields\":[{\"name\":\"errorMessage\",\"type\":[\"string\",\"null\"]}]}");
var schema = Schema.Parse($"[{columnCountSchema},{errorMessageSchema}]");
Console.WriteLine(schema.ToString());

The output:

{"type":[{"type":"record","name":"columnCount","namespace":"avro.test.header","fields":[{"name":"columnCount","type":"int"}]},{"type":"record","name":"errorMessage","namespace":"avro.test.header","fields":[{"name":"errorMessage","type":["string","null"]}]}]}

 

When I use Avro in C# to parse this union schema JSON string, it is correct. Schema.ParseJson() in C#

But when I use Avro in Java to parse the union schema in Stream from C#, there is the following error message:

org.apache.avro.SchemaParseException: No type: {"type":[{"type":"record","name":"columnCount","namespace":"avro.test.header","fields":[{"name":"columnCount","type":"int"}]},{"type":"record","name":"errorMessage","namespace":"avro.test.header","fields":[{"name":"errorMessage","type":["string","null"]}]}]}org.apache.avro.SchemaParseException: No type: {"type":[{"type":"record","name":"columnCount","namespace":"avro.test.header","fields":[{"name":"columnCount","type":"int"}]},{"type":"record","name":"errorMessage","namespace":"avro.test.header","fields":[{"name":"errorMessage","type":["string","null"]}]}]} at org.apache.avro.Schema.getRequiredText(Schema.java:1784) at org.apache.avro.Schema.parse(Schema.java:1637) at org.apache.avro.Schema$Parser.parse(Schema.java:1425) at org.apache.avro.Schema$Parser.parse(Schema.java:1413) at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:130) at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:90) at avro.AvroService.read(AvroService.java:68) at avro.AvroService$1.onResponse(AvroService.java:144) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

I think this format of union schema should be supported in Java.

 

Issue Links

• links to: GitHub Pull Request #1299
• links to: GitHub Pull Request #1303
• links to: GitHub Pull Request #1303