Currently the base types in Atlas do not include Avro schemas. It would be nice to add typedef for Avro schema and any associated metadata to support schema evolution.
- For example, Avro_schema type supports:
- All avro types, both primitive and complex, including union types, as fields of schema
- All types have doc strings and defaults
- A field of a schema can be another schema
- Indefinite nesting of records, arrays.
- Associated entities array attribute contains pointers to all datasets that reflect the avro schema
- Fully expanded avroNotation for use in serDe
- Schema evolution features such as isLatest (Boolean) and version number
- Schema evolution Process
- Input: avro schema
- Output: new version of avro schema
- Compatibility: FULL, BACKWARD, FORWARD, NONE
- IsBreakingChange (Boolean): does the change produce an incompatible schema? (ie its compatibility is not “FULL”)