[CALCITE-429] Add statistics SPI for lattice optimization algorithm

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0-incubating
• Assignee: Julian Hyde
• Reporter: Julian Hyde
• Created: 30/Sep/14 20:45
• Updated: 01/Sep/15 03:02
• Resolved: 13/Jun/15 01:15

Description

In OPTIQ-427 we added an an optimization algorithm to choose an initial set of tiles to materialize. The rowCountEstimate attribute is the number of rows in the lattice.

Add an SPI to generate estimates of (a) the number of rows in the lattice, (b) the number of rows in a given tile of the lattice (specified by its dimensions).

Also add a default implementation of the SPI that executes SQL queries, caches the results, and uses some kind of approximation for the cardinalities of sets of attributes e.g. if quarter has 4 distinct values and year has 10 distinct values then (year, quarter) has 40 distinct values (or perhaps an expectation of 39.79 distinct values in a table with 3,650 rows, per the formula n . (1 - ((n - 1) / n) ^ p)).

Implementations that read stats from external stats tables, or execute SQL that samples a small percentage of the rows, would also be possible.

Issue Links

• is duplicated by: CALCITE-711 Cardinality provider for use by lattice algorithm