Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-2003

Report specific location of schema incompatibilities

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.8.1
    • Fix Version/s: 1.9.0
    • Component/s: java
    • Labels:
      None
    • Environment:

      Any java env

    • Release Note:
      SchemaCompatibility reports location of incompatibility as a JSON Pointer.

      Description

      Overview

      Building on the work to improve schema incompatibility reporting in AVRO-1933, it would be useful if the SchemaCompatibility classes could also report the location in the schema where any incompatibility was encountered.

      It is recommended that the location reported is both easily readable by humans and machines. In the first case this would assist schema developers to pin-point issues in there schema documents, and in the latter case it provides a useful mechanism to schema tooling, such as IDEs and editors, to easily select the pertinent nodes in the Schema document tree.

      Implementation specifics

      To meet this requirements it is suggested that the location is encoded using the JSON Pointer specification. This is both easily parsed by users, but is also supported by a number of libraries for a range of common programming languages and platforms.

      Examples

      Given the following example schema, consider some incompatibility scenarios. For each case an expected JSON Pointer description of the incompatibility location is described:

      {
        "type": "record",
        "name": "myRecord",
        "fields" : [
          {"name": "pField", "type": "long"},
          {"name": "uField", "type":
            ["null", "int", "string"]
          },
          {"name": "eField", "type": 
            { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS", "DIAMONDS", "CLUBS"] }
          },
          {"name": "aField", "type":
            {"type": "array", "items": "string"}
          },
          {"name": "mField", "type": 
            {"type": "map", "values": "long"}
          },
          {"name": "fField", "type": 
            {"type": "fixed", "size": 16, "name": "md5"}
          }
        ]
      }
      

      Possible incompatibility scenarions and the location that would be reported back to the user/tool:

      • Root type incompatibility; report location: /
      • Record name mismatch; report location: /name
      • pField type incompatibility; report location: /fields/0/type
      • uField field type incompatibility; report location: /fields/1/type
      • uField missing union branch string; report location: /fields/1/type/2
      • eField field type incompatibility; report location: /fields/2/type
      • eField missing enum symbol; report location: /fields/2/type/symbols
      • eField enum name mismatch; report location: /fields/2/type/name
      • aField field type incompatibility; report location: /fields/3/type
      • aField array element type incompatibility; report location: /fields/3/type/items
      • mField field type incompatibility; report location: /fields/4/type
      • mField map value type incompatibility; report location: /fields/4/type/values
      • fField field type incompatibility; report location: /fields/5/type
      • fField fixed name mismatch; report location: /fields/5/type/name
      • fField fixed size type incompatibility; report location: /fields/5/type/size
      • fField missing default value; report location: /fields/5

      Notes

      • This ticket depends on AVRO-1933 and associated patches.
      1. AVRO-2003.patch
        47 kB
        Elliot West

        Issue Links

          Activity

          Hide
          satish.duggana Satish Duggana added a comment -

          Elliot West Can you raise PR by taking changes from AVRO-1933 PR

          Show
          satish.duggana Satish Duggana added a comment - Elliot West Can you raise PR by taking changes from AVRO-1933 PR
          Hide
          teabot Elliot West added a comment - - edited

          Satish Duggana, should I fork epkanol/avro-1:AVRO-1933 and create a PR to merge back into there, or fork apache/avro:master and pull the AVRO-1933 changes into there to create a single über PR?

          PR is located here: https://github.com/apache/avro/pull/201

          Show
          teabot Elliot West added a comment - - edited Satish Duggana , should I fork epkanol/avro-1: AVRO-1933 and create a PR to merge back into there, or fork apache/avro:master and pull the AVRO-1933 changes into there to create a single über PR? PR is located here: https://github.com/apache/avro/pull/201
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user teabot opened a pull request:

          https://github.com/apache/avro/pull/201

          AVRO-2003: Report specific location of schema incompatibilities

          Includes patch AVRO-1933(https://issues.apache.org/jira/browse/AVRO-1933) whose PR is located here: #200

          JIRA: AVRO-2003(https://issues.apache.org/jira/browse/AVRO-2003)

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/HotelsDotCom/avro AVRO-2003

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/avro/pull/201.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #201


          commit f9bf0308fbb83f25b9e87f10f0ab16dfae93e845
          Author: Anders Sundelin <anders.sundelin@ericsson.com>
          Date: 2017-03-02T10:12:03Z

          AVRO-1933: Add more specific error details to SchemaCompatibility class

          commit 7015f9f71ebc789129e8811c9c43ba103be0bf68
          Author: Elliot West <ewest@hotels.com>
          Date: 2017-03-03T10:58:48Z

          Track location of incompatibility using a JSON Pointer.
          Additional test cases for nested incompatibility locations.
          msg -> message
          details -> result


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user teabot opened a pull request: https://github.com/apache/avro/pull/201 AVRO-2003 : Report specific location of schema incompatibilities Includes patch AVRO-1933 ( https://issues.apache.org/jira/browse/AVRO-1933 ) whose PR is located here: #200 JIRA: AVRO-2003 ( https://issues.apache.org/jira/browse/AVRO-2003 ) You can merge this pull request into a Git repository by running: $ git pull https://github.com/HotelsDotCom/avro AVRO-2003 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/avro/pull/201.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #201 commit f9bf0308fbb83f25b9e87f10f0ab16dfae93e845 Author: Anders Sundelin <anders.sundelin@ericsson.com> Date: 2017-03-02T10:12:03Z AVRO-1933 : Add more specific error details to SchemaCompatibility class commit 7015f9f71ebc789129e8811c9c43ba103be0bf68 Author: Elliot West <ewest@hotels.com> Date: 2017-03-03T10:58:48Z Track location of incompatibility using a JSON Pointer. Additional test cases for nested incompatibility locations. msg -> message details -> result
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 compile 0m 0s java in the patch failed.
          +1 buildtest 6m 39s master passed
          -1 javadoc 0m 2s java in the patch failed.
          -1 compile 0m 1s java in the patch failed.
          -1 javac 0m 1s java in the patch failed.
          -1 whitespace 0m 0s The patch has 21 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 buildtest 6m 21s the patch passed
          -1 javadoc 0m 2s java in the patch failed.
          13m 37s



          Subsystem Report/Notes
          Docker Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a
          JIRA Issue AVRO-2003
          GITHUB PR https://github.com/apache/avro/pull/201
          Optional Tests buildtest javac compile javadoc
          uname Linux 5ed3aa5d1014 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux
          Build tool build
          git revision master / 793178a
          Default Java 1.7.0_111
          compile https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt
          javadoc https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/javadoc--lang_java.txt
          compile https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt
          javac https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt
          whitespace https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/javadoc--lang_java.txt
          modules C: lang/java U: lang/java
          Console output https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/console
          Powered by Apache Yetus 0.4.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 compile 0m 0s java in the patch failed. +1 buildtest 6m 39s master passed -1 javadoc 0m 2s java in the patch failed. -1 compile 0m 1s java in the patch failed. -1 javac 0m 1s java in the patch failed. -1 whitespace 0m 0s The patch has 21 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 buildtest 6m 21s the patch passed -1 javadoc 0m 2s java in the patch failed. 13m 37s Subsystem Report/Notes Docker Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a JIRA Issue AVRO-2003 GITHUB PR https://github.com/apache/avro/pull/201 Optional Tests buildtest javac compile javadoc uname Linux 5ed3aa5d1014 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux Build tool build git revision master / 793178a Default Java 1.7.0_111 compile https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt javadoc https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/javadoc--lang_java.txt compile https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt javac https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/compile--lang_java.txt whitespace https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/artifact/patchprocess/javadoc--lang_java.txt modules C: lang/java U: lang/java Console output https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/14/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 buildtest 0m 0s master passed
          -1 whitespace 0m 0s The patch has 21 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 buildtest 6m 40s the patch passed
          7m 5s



          Subsystem Report/Notes
          Docker Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a
          JIRA Issue AVRO-2003
          GITHUB PR https://github.com/apache/avro/pull/201
          Optional Tests buildtest javac
          uname Linux 8a7da17fc74b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux
          Build tool build
          git revision master / 793178a
          Default Java 1.7.0_111
          whitespace https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/15/artifact/patchprocess/whitespace-eol.txt
          modules C: lang/java U: lang/java
          Console output https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/15/console
          Powered by Apache Yetus 0.4.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 buildtest 0m 0s master passed -1 whitespace 0m 0s The patch has 21 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 buildtest 6m 40s the patch passed 7m 5s Subsystem Report/Notes Docker Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a JIRA Issue AVRO-2003 GITHUB PR https://github.com/apache/avro/pull/201 Optional Tests buildtest javac uname Linux 8a7da17fc74b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 GNU/Linux Build tool build git revision master / 793178a Default Java 1.7.0_111 whitespace https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/15/artifact/patchprocess/whitespace-eol.txt modules C: lang/java U: lang/java Console output https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/15/console Powered by Apache Yetus 0.4.0 http://yetus.apache.org This message was automatically generated.
          Show
          sacharya Suraj Acharya added a comment - Ignore build 14. Build 15 artifact : https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/ws/component/patchprocess/whitespace-eol.txt
          Hide
          teabot Elliot West added a comment -

          Please see this mail list post for a workaround:
          http://mail-archives.apache.org/mod_mbox/avro-user/201707.mbox/%3cCAC3gpCbNGSNZ744v7scST9n+6x7J4-SML=mwhcRNt176tgTFcw@mail.gmail.com%3e

          I'll attempt to resolve the whitespace issue described above.

          Show
          teabot Elliot West added a comment - Please see this mail list post for a workaround: http://mail-archives.apache.org/mod_mbox/avro-user/201707.mbox/%3cCAC3gpCbNGSNZ744v7scST9n+6x7J4-SML=mwhcRNt176tgTFcw@mail.gmail.com%3e I'll attempt to resolve the whitespace issue described above.
          Hide
          teabot Elliot West added a comment -

          Suraj Acharya, Satish Duggana, I believe that I've now resolved the whitespace issue that was causing the -1 build. I've pushed the changes to the PR on GitHub, thanks.

          Show
          teabot Elliot West added a comment - Suraj Acharya , Satish Duggana , I believe that I've now resolved the whitespace issue that was causing the -1 build. I've pushed the changes to the PR on GitHub, thanks.
          Hide
          nkollar Nandor Kollar added a comment -

          Elliot West now AVRO-1933 is committed to master, due to this your PR has several merge conflicts. Could you please update it with merging your changes to latest Apache master?

          Show
          nkollar Nandor Kollar added a comment - Elliot West now AVRO-1933 is committed to master, due to this your PR has several merge conflicts. Could you please update it with merging your changes to latest Apache master?
          Hide
          teabot Elliot West added a comment -

          Nandor Kollar certainly, I'll look at this shortly.

          Show
          teabot Elliot West added a comment - Nandor Kollar certainly, I'll look at this shortly.
          Hide
          teabot Elliot West added a comment -

          Nandor Kollar, I believe this is ready for review. Apologies for the stream of small commits; I had to back out a bunch of formatting changes that were perhaps introduced from an early AVRO-1933 patch (I think...). Anyway, I believe the PR now includes only changes related to this feature. Thank you.

          Show
          teabot Elliot West added a comment - Nandor Kollar , I believe this is ready for review. Apologies for the stream of small commits; I had to back out a bunch of formatting changes that were perhaps introduced from an early AVRO-1933 patch (I think...). Anyway, I believe the PR now includes only changes related to this feature. Thank you.
          Hide
          nkollar Nandor Kollar added a comment -

          Thanks Elliot West, I'll help the committers with a review soon!

          Show
          nkollar Nandor Kollar added a comment - Thanks Elliot West , I'll help the committers with a review soon!

            People

            • Assignee:
              teabot Elliot West
              Reporter:
              teabot Elliot West
            • Votes:
              3 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development