[AVRO-1915] AvroTypeException decoding from earlier schema version

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.7.7
• Component/s: java
• Assignee: Unassigned
• Reporter: NPE
• Created: 14/Sep/16 12:40
• Updated: 21/Sep/16 17:08
• Resolved: 21/Sep/16 17:08

Description

We have two services which communicate with one another by sending JSON-encoded Avro-based messages over Kafka. We want to update the schema for messages sent from service A to service B by adding an additional string field with a default value of "" (empty string). We have tested by initially adding the updated schema to service B (the reader) and continuing to send messages in the older format from service A (the writer).

Simplified example of old schema (some fields omitted):

{
	"type": "record",
	"name": "Envelope",
	"fields": [{
		"name": "appId",
		"type": "string"
	}, {
		"name": "time",
		"type": "long"
	}, {
		"name": "type",
		"type": "string"
	}, {
		"name": "payload",
		"type": [{
			"type": "record",
			"name": "MessagePayload",
			"fields": [{
				"name": "context",
				"type": {
					"type": "record",
					"name": "PayloadContext",
					"fields": [{
						"name": "source",
						"type": "string"
					}, {
						"name": "requestId",
						"type": "string"
					}]
				}
			}, {
				"name": "content",
				"type": "string"
			}, {
				"name": "contentType",
				"type": "string"
			}]
		}]
	}]
}

Simplified example of new schema (some fields omitted):

{
	"type": "record",
	"name": "Envelope",
	"fields": [{
		"name": "appId",
		"type": "string"
	}, {
		"name": "time",
		"type": "long"
	}, {
		"name": "type",
		"type": "string"
	}, {
		"name": "payload",
		"type": [{
			"type": "record",
			"name": "MessagePayload",
			"fields": [{
				"name": "context",
				"type": {
					"type": "record",
					"name": "PayloadContext",
					"fields": [{
						"name": "source",
						"type": "string"
					}, {
						"name": "requestId",
						"type": "string"
					}, {
						"name": "newField",
						"type": "string",
						"default": ""
					}]
				}
			}, {
				"name": "content",
				"type": "string"
			}, {
				"name": "contentType",
				"type": "string"
			}]
		}]
	}]
}

Our understanding was that the reader, with the newer schema, should be able to parse messages sent with the older given the default value for the missing field; however, we are getting the following exception:

org.apache.avro.AvroTypeException: Expected string. Got END_OBJECT

Are we missing something here?