[AVRO-2003] Report specific location of schema incompatibilities

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.8.1
• Fix Version/s: 1.9.0
• Component/s: java

Environment

Any java env

• Hadoop Flags: Incompatible change
• Release Note: SchemaCompatibility reports location of incompatibility as a JSON Pointer. This is a backward incompatible change, because the public interface of SchemaCompatibility class changed.
• Assignee: Elliot West
• Reporter: Elliot West
• Created: 23/Feb/17 12:01
• Updated: 05/Dec/17 09:02
• Resolved: 05/Dec/17 09:01

Description

Overview

Building on the work to improve schema incompatibility reporting in AVRO-1933, it would be useful if the SchemaCompatibility classes could also report the location in the schema where any incompatibility was encountered.

It is recommended that the location reported is both easily readable by humans and machines. In the first case this would assist schema developers to pin-point issues in there schema documents, and in the latter case it provides a useful mechanism to schema tooling, such as IDEs and editors, to easily select the pertinent nodes in the Schema document tree.

Implementation specifics

To meet this requirements it is suggested that the location is encoded using the JSON Pointer specification. This is both easily parsed by users, but is also supported by a number of libraries for a range of common programming languages and platforms.

Examples

Given the following example schema, consider some incompatibility scenarios. For each case an expected JSON Pointer description of the incompatibility location is described:

{
  "type": "record",
  "name": "myRecord",
  "fields" : [
    {"name": "pField", "type": "long"},
    {"name": "uField", "type":
      ["null", "int", "string"]
    },
    {"name": "eField", "type": 
      { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS", "DIAMONDS", "CLUBS"] }
    },
    {"name": "aField", "type":
      {"type": "array", "items": "string"}
    },
    {"name": "mField", "type": 
      {"type": "map", "values": "long"}
    },
    {"name": "fField", "type": 
      {"type": "fixed", "size": 16, "name": "md5"}
    }
  ]
}

Possible incompatibility scenarions and the location that would be reported back to the user/tool:

• Root type incompatibility; report location: /
• Record name mismatch; report location: /name
• pField type incompatibility; report location: /fields/0/type
• uField field type incompatibility; report location: /fields/1/type
• uField missing union branch string; report location: /fields/1/type/2
• eField field type incompatibility; report location: /fields/2/type
• eField missing enum symbol; report location: /fields/2/type/symbols
• eField enum name mismatch; report location: /fields/2/type/name
• aField field type incompatibility; report location: /fields/3/type
• aField array element type incompatibility; report location: /fields/3/type/items
• mField field type incompatibility; report location: /fields/4/type
• mField map value type incompatibility; report location: /fields/4/type/values
• fField field type incompatibility; report location: /fields/5/type
• fField fixed name mismatch; report location: /fields/5/type/name
• fField fixed size type incompatibility; report location: /fields/5/type/size
• fField missing default value; report location: /fields/5

Notes

• This ticket depends on AVRO-1933 and associated patches.

Issue Links

• depends upon: AVRO-1931 SchemaCompatibility fails to recognize reader compatible with all branches of a union
• depends upon: AVRO-1933 SchemaCompatibility class could be more user-friendly about incompatibilities
• links to: GitHub Pull Request #201