[IMPALA-4854] COMPUTE INCREMENTAL STATS should ignore missing stats on complex columns

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.9.0
• Component/s: Frontend
• Labels: compute-stats ramp-up testability
• Target Version: Impala 2.9.0
• Assignee: Alexander Behm
• Reporter: Nathan Salmon
• Created: 01/Feb/17 18:18
• Updated: 17/Feb/17 06:13
• Resolved: 17/Feb/17 06:13

Description

After executing "compute incremental stats" on a table, by design, future calls to "compute incremental stats" only compute stats for partitions for which there are no statistics. However, when statistics are found to be missing for a column, e.g. a column was added to the schema since the last computation, incremental stats will be recomputed for all partitions. Impala doesn't currently compute statistics for complex columns, such as arrays and structs. Because of this, stats for these types of columns are always found to be missing, which incorrectly causes stats to be re-computed for all partitions on every run. Missing stats for complex columns on previously stat-computed partitions should be ignored when determining if re-computation is necessary, as re-computing stats will never remedy this situation.