[IMPALA-4248] Refresh needed to pick up new Avro schema modified from inside impala

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.6.0
• Component/s: Frontend
• Labels: avro ramp-up usability
• Assignee: Unassigned
• Reporter: Balazs Jeszenszky
• Created: 05/Oct/16 08:46
• Updated: 11/Jul/17 08:52

Description

Avro schema from within Impala can be changed by doing:

ALTER TABLE avro_table SET TBLPROPERTIES('avro.schema.url'='hdfs:///tmp/evolved.avsc');
ALTER TABLE avro_table ADD COLUMNS(c3 string);

However, if the order of the statements is reversed, Impala also needs a refresh:

ALTER TABLE avro_table ADD COLUMNS(c3 string);
ALTER TABLE avro_table SET TBLPROPERTIES('avro.schema.url'='hdfs:///tmp/evolved.avsc');
compute stats avro_table;
---->ERROR: AnalysisException: Cannot COMPUTE STATS on Avro table 'avro_table' because its column definitions do not match those in the Avro schema.
Missing Avro-schema column corresponding to column definition 'c3' of type 'string' at position '1'.
Please re-create the table with column definitions, e.g., using the result of 'SHOW CREATE TABLE'

refresh avro_table;
compute stats avro_table;
+-----------------------------------------+
| summary                                 |
+-----------------------------------------+
| Updated 1 partition(s) and 3 column(s). |
+-----------------------------------------+

Issue Links

• is related to: IMPALA-5647 Allow COMPUTE STATS after changing the Avro schema in the schema file