Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-2003

Report specific location of schema incompatibilities

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.8.1
    • Fix Version/s: 1.9.0
    • Component/s: java
    • Labels:
      None
    • Environment:

      Any java env

    • Hadoop Flags:
      Incompatible change
    • Release Note:
      SchemaCompatibility reports location of incompatibility as a JSON Pointer. This is a backward incompatible change, because the public interface of SchemaCompatibility class changed.

      Description

      Overview

      Building on the work to improve schema incompatibility reporting in AVRO-1933, it would be useful if the SchemaCompatibility classes could also report the location in the schema where any incompatibility was encountered.

      It is recommended that the location reported is both easily readable by humans and machines. In the first case this would assist schema developers to pin-point issues in there schema documents, and in the latter case it provides a useful mechanism to schema tooling, such as IDEs and editors, to easily select the pertinent nodes in the Schema document tree.

      Implementation specifics

      To meet this requirements it is suggested that the location is encoded using the JSON Pointer specification. This is both easily parsed by users, but is also supported by a number of libraries for a range of common programming languages and platforms.

      Examples

      Given the following example schema, consider some incompatibility scenarios. For each case an expected JSON Pointer description of the incompatibility location is described:

      {
        "type": "record",
        "name": "myRecord",
        "fields" : [
          {"name": "pField", "type": "long"},
          {"name": "uField", "type":
            ["null", "int", "string"]
          },
          {"name": "eField", "type": 
            { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS", "DIAMONDS", "CLUBS"] }
          },
          {"name": "aField", "type":
            {"type": "array", "items": "string"}
          },
          {"name": "mField", "type": 
            {"type": "map", "values": "long"}
          },
          {"name": "fField", "type": 
            {"type": "fixed", "size": 16, "name": "md5"}
          }
        ]
      }
      

      Possible incompatibility scenarions and the location that would be reported back to the user/tool:

      • Root type incompatibility; report location: /
      • Record name mismatch; report location: /name
      • pField type incompatibility; report location: /fields/0/type
      • uField field type incompatibility; report location: /fields/1/type
      • uField missing union branch string; report location: /fields/1/type/2
      • eField field type incompatibility; report location: /fields/2/type
      • eField missing enum symbol; report location: /fields/2/type/symbols
      • eField enum name mismatch; report location: /fields/2/type/name
      • aField field type incompatibility; report location: /fields/3/type
      • aField array element type incompatibility; report location: /fields/3/type/items
      • mField field type incompatibility; report location: /fields/4/type
      • mField map value type incompatibility; report location: /fields/4/type/values
      • fField field type incompatibility; report location: /fields/5/type
      • fField fixed name mismatch; report location: /fields/5/type/name
      • fField fixed size type incompatibility; report location: /fields/5/type/size
      • fField missing default value; report location: /fields/5

      Notes

      • This ticket depends on AVRO-1933 and associated patches.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                teabot Elliot West
                Reporter:
                teabot Elliot West
              • Votes:
                3 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: