Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-2694

Avro schema typedef and support for Avro schema evolution in Atlas

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: atlas-core
    • Labels:
      None

      Description

      Currently the base types in Atlas do not include Avro schemas. It would be nice to add typedef for Avro schema and any associated metadata to support schema evolution.

      • For example, Avro_schema type supports:
        • All avro types, both primitive and complex, including union types, as fields of schema
        • All types have doc strings and defaults
        • A field of a schema can be another schema
        • Indefinite nesting of records, arrays.
        • Associated entities array attribute contains pointers to all datasets that reflect the avro schema
        • Fully expanded avroNotation for use in serDe
        • Schema evolution features such as isLatest (Boolean) and version number
      • Schema evolution Process
        • Input: avro schema
        • Output: new version of avro schema
        • Compatibility: FULL, BACKWARD, FORWARD, NONE
        • IsBreakingChange (Boolean): does the change produce an incompatible schema? (ie its compatibility is not “FULL”)
          *

        Attachments

        1. ATLAS-2694.patch
          8 kB
          Madhan Neethiraj
        2. ATLAS-2694-2.patch
          3 kB
          Madhan Neethiraj
        3. avro_atlas_types_08.json
          5 kB
          Barbara Eckman

          Activity

            People

            • Assignee:
              barbara Barbara Eckman
              Reporter:
              srikvenk Srikanth Venkat
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: