[IMPALA-3722] Avro codegen can be unnecessarily disabled

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.6.0
• Component/s: Backend
• Labels: avro codegen ramp-up
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Skye Wanderman-Milne
• Created: 10/Jun/16 18:22
• Updated: 19/Jun/20 19:22

Description

We use avro_schema_equal() from the Avro C library to determine if a file's schema matches the table schema, and if they don't match we disable codegen for that file (https://github.com/cloudera/Impala/blob/cdh5-trunk/be/src/exec/hdfs-avro-scanner.cc#L153). However, avro_schema_equal() is unnecessarily restrictive, because it compares the records' names and namespaces, which don't have to be the same to enable codegen. There are probably other checks we don't need as well, e.g. default values. We should write our own schema comparison function that is tailored to what must match for codegen specifically.