Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-429

Add statistics SPI for lattice optimization algorithm

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      In OPTIQ-427 we added an an optimization algorithm to choose an initial set of tiles to materialize. The rowCountEstimate attribute is the number of rows in the lattice.

      Add an SPI to generate estimates of (a) the number of rows in the lattice, (b) the number of rows in a given tile of the lattice (specified by its dimensions).

      Also add a default implementation of the SPI that executes SQL queries, caches the results, and uses some kind of approximation for the cardinalities of sets of attributes e.g. if quarter has 4 distinct values and year has 10 distinct values then (year, quarter) has 40 distinct values (or perhaps an expectation of 39.79 distinct values in a table with 3,650 rows, per the formula n . (1 - ((n - 1) / n) ^ p)).

      Implementations that read stats from external stats tables, or execute SQL that samples a small percentage of the rows, would also be possible.

        Issue Links

          Activity

          Show
          julianhyde Julian Hyde added a comment - Work started in https://github.com/julianhyde/incubator-calcite/tree/429-lattice-stats-spi .
          Show
          julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/4cc539fc .
          Hide
          jnadeau Jacques Nadeau added a comment -

          Resolved in release 1.4.0-incubating (2015-08-23)

          Show
          jnadeau Jacques Nadeau added a comment - Resolved in release 1.4.0-incubating (2015-08-23)

            People

            • Assignee:
              julianhyde Julian Hyde
              Reporter:
              julianhyde Julian Hyde
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development