[IMPALA-1570] DROP / COMPUTE incremental stats with dynamic partition specs

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.1
• Fix Version/s: Impala 2.8.0
• Component/s: Frontend
• Labels: compute-stats ramp-up
• Target Version: Product Backlog
• Assignee: Amos Bird
• Reporter: Henry Robinson
• Created: 04/Dec/14 01:22
• Updated: 04/Jan/18 16:54
• Resolved: 17/Nov/16 01:40

Description

COMPUTE INCREMENTAL STATS, and its counterpart DROP INCREMENTAL STATS can both take PARTITION ... clauses to specify the precise partition to act on. If a small set of partitions need updating (or if the user wants to batch incremental stat computations in clumps), it would be good to allow dynamic partition specs, and only drop or update those partitions matched by the looser specification.

For example COMPUTE INCREMENTAL STATS tbl PARTITION(year=2009, month) would update all months in 2009 that were missing incremental stats.

For maximum benefit, we could add a set of constraints:

COMPUTE INCREMENTAL STATS tbl PARTITION(year, month) WHERE year > 2009 and month=10

would update all the October partitions since 2010 inclusive.

We already have logic to do partition pruning in the frontend - this is the reverse. COMPUTE INCREMENTAL STATS already is able to work on a subset of partitions, so the work here should be substantially in the frontend.