[IMPALA-867] Fail early (in analysis) when COMPUTE STATS is run against Avro table with no columns

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.2.4, Impala 1.3
• Fix Version/s: Impala 1.3
• Assignee: Alexander Behm
• Reporter: Lenni Kuff
• Created: 10/Mar/14 21:57
• Updated: 23/Jul/14 17:24
• Resolved: 15/Mar/14 04:48

Description

If an Avro table is created without any columns defined (eg. the schema is only defined in the table's avro.schema TABLE/SERDEPROPERTIES), then COMPUTE STATS will fail with a somewhat cryptic error:

InternalException: Column 'col_name' for which stats gathering is requested doesn't exist.

This is because of a bug in the Hive Metastore which should reject creating tables without any column definitions.

However, this error is returned at the very end of the COMPUTE STATS operation, which may take a long time for large tables. We should see if it is possible to detect the invalid schema early, and fail COMPUTE STATS in analysis.

We should also verify whether COMPUTE STATS works in the case where the table contains some columns, but they don't match the avro schema. For example:

create table (int_col int)
...
WITH SERDEPROPERTIES ('avro.schema.literal'='{
"name": "a",
"type": "record",
"fields": [
  {"name":"string_col",  "type": ["null", "string"],  "default": null}
]}')