[ATLAS-2694] Avro schema typedef and support for Avro schema evolution in Atlas

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: atlas-core
• Assignee: Barbara Eckman
• Reporter: Srikanth Venkat
• Created: 16/May/18 21:21
• Updated: 29/Apr/21 05:09
• Resolved: 25/May/18 21:28

Description

Currently the base types in Atlas do not include Avro schemas. It would be nice to add typedef for Avro schema and any associated metadata to support schema evolution.

• For example, Avro_schema type supports:
  • All avro types, both primitive and complex, including union types, as fields of schema
  • All types have doc strings and defaults
  • A field of a schema can be another schema
  • Indefinite nesting of records, arrays.
  • Associated entities array attribute contains pointers to all datasets that reflect the avro schema
  • Fully expanded avroNotation for use in serDe
  • Schema evolution features such as isLatest (Boolean) and version number
• Schema evolution Process
  • Input: avro schema
  • Output: new version of avro schema
  • Compatibility: FULL, BACKWARD, FORWARD, NONE
  • IsBreakingChange (Boolean): does the change produce an incompatible schema? (ie its compatibility is not “FULL”)
    *