[DRILL-4548] Avro fails on schema changes even when selecting from a single file (Union related)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0, 1.7.0
• Component/s: Storage - Avro
• Assignee: Unassigned
• Reporter: Stefán Baxter
• Created: 28/Mar/16 20:51
• Updated: 28/Mar/16 22:33

Description

Hi,

I have reworked/refactored our Avro based logging system trying to make the whole Drill + Avro->Parquet experience a bit more agreeable.

Long story short I'm getting this error when selecting form multiple Avro files even though these files share the EXCACT same schema:

Error: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema changes
Fragment 0:0
[Error Id: 00d49aa2-5564-497e-a330-e852d5889beb on swift:31010] (state=,code=0)

We are using union types but only to allow for null values as seems to be supported by drill as per this comment in the Drill code:
// currently supporting only nullable union (optional fields) like ["null", "some-type"].

This happens for a very simple group_by + count query that only uses two fields in Avro and neither one of them uses a Union construct so and both of them contain string values in every case.

I now think this has nothing to do with the union types since the query uses only simple string, unless there is a full schema validation done on the content of the files rather then the identical Avro schema embedded in both files.