Details
-
New Feature
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
1.7.6
-
None
-
None
Description
There are a few use cases for this:
Various Avro input data to one common output
You want to pickup Avro files in different schemas and normalize into one. You might wish to transform to the superset of the input schemas.
Aggregating Raw Data
You want to rewrite data grouped by some fields and aggregated. The output Avro in this case would be a subset of the input Avro, where at least the group by fields are in both input and output schemas.
Alternate Views
You have Avro data that you want to trim different ways to create subsets that would be useful for views in Hive or exports for SQL tables.
Schema Migration
You've added fields to a schema and you are storing data in both the old and new schemas. You have Avro in an old schema and you can't process it with Avro in the new schema (using pig or java map-reduce). AutoMapping would up-convert your old data by setting null for the new fields added, and all data are in the new schema. This was asked about on StackOverflow.
Considerations:
- Loop over the source schema fields available to automap over and return any that were unable to be mapped.
- Allow mappings between compatible types. For example going from integers to longs, floats to strings, etc.
- Field names match case-sensitive.
- Make use of aliases in the schema when considering fields to automap.
- Deep copy nested structures like arrays and maps