[DRILL-7398] dfs avro file support to treat non existent fields as null

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Storage - Avro
• Assignee: Unassigned
• Reporter: Dan Schmitt
• Created: 09/Oct/19 18:38
• Updated: 23/Jan/20 15:54
• Resolved: 23/Jan/20 15:54

Description

It would be nice to have an option to return null for non existing fields.

Currently we have a use case of evolving avro file schemas where things get added.

It would be nice to be able to query old and new data without having to rewrite the old data with a new schema with null/optional/default values.

A short term fix would be to provide an option to return null with dfs for avro fields that aren't found in a file.

A nicer fix would be to support a resolving schema that can be given to dfs to read the avro files to present a uniform interface (so aliases, defaults, etc. could be utilized without rewriting each avro file.)

Issue Links

• is part of: DRILL-7454 Convert the Avro format plugin to use EVF