[AVRO-3027] .net consumer Backward compatibility issue.

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.10.1
• Component/s: csharp, java
• Labels: pull-request-available

Environment

.Net core 3.1

• Assignee: Unassigned
• Reporter: Yevhen Cherkes
• Created: 19/Jan/21 14:47
• Updated: 28/Aug/22 08:15

Description

We are facing the following exception on the .net core consumer's side:
(Java-based consumers work with no issues for this particular case.)

No matching schema for {"type":"record","name":"InnerType1","namespace":"com.mycorp.mynamespace","fields":[

{"name":"value","type":"string"}

]} in {"type":["null",{"type":"record","name":"InnerType2","namespace":"com.mycorp.mynamespace","fields":[

{"name":"value","type":"string"}

]}]}

at Avro.Generic.DefaultReader.findBranch(UnionSchema us, Schema s)
 at Avro.Generic.DefaultReader.ReadUnion(Object reuse, UnionSchema writerSchema, Schema readerSchema, Decoder d)
 at Avro.Generic.DefaultReader.Read(Object reuse, Schema writerSchema, Schema readerSchema, Decoder d)
 at Avro.Specific.SpecificDefaultReader.ReadRecord(Object reuse, RecordSchema writerSchema, Schema readerSchema, Decoder dec)

We have 2 versions of the schema on a topic:

Schema v1:

{
    "fields": [
      {
        "name": "InnerComplexValue",
        "type": [
          "null",
          {
			"fields": [
			  {
				"name": "value",
				"type": "string"
			  }
			],
			"name": "InnerType1",
			"namespace": "com.mycorp.mynamespace",
			"type": "record"
		  }
        ]
      }
    ],
    "name": "RootType",
    "namespace": "com.mycorp.mynamespace",
    "type": "record"
  }

 Schema v2:

{
    "fields": [
      {
        "name": "InnerComplexValue",
        "type": [
          "null",
          {
			"fields": [
			  {
				"name": "value",
				"type": "string"
			  }
			],
			"name": "InnerType2",
			"namespace": "com.mycorp.mynamespace",
			"type": "record"
		  }
        ]
      }
    ],
    "name": "RootType",
    "namespace": "com.mycorp.mynamespace",
    "type": "record"
  }

 InnerType1 -> InnerType2 is the only change.

The schema for a topic is configured with Backward compatibility.

An updated version of a Schema is saved with no errors.

Then we generated a specific record RootType with avrogen tool.

Reproduction code:
 

var base64 = "AhR0ZXN0IHZhbHVl";
using var stream = new MemoryStream(Convert.FromBase64String(base64));

var schema_v1 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
                                                 "[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType1\"" +
                                                 ",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");

var schema_v2 = Schema.Parse("{\"type\":\"record\",\"name\":\"RootType\",\"namespace\":\"com.mycorp.mynamespace\",\"fields\":" +
                                         "[{\"name\":\"InnerComplexValue\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"InnerType2\"" +
                                         ",\"fields\":[{\"name\":\"value\",\"type\":\"string\"}]}]}]}");

var specificRecord = Deserialize<RootType>(stream, schema_v1, schema_v2);

Full sources see in attachments.

Method Deserialize was taken from the SpecificTests.cs#L408-L417

 

Probably, the reason is in a different Schema resolution code:
java: https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Resolver.java#L647-L659
.net: https://github.com/apache/avro/blob/master/lang/csharp/src/apache/main/Schema/RecordSchema.cs#L310-L312

Issue Links

• links to: GitHub Pull Request #1103