Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26313

Aggregate all column statistics into a single field in metastore

    XMLWordPrintableJSON

Details

    Description

      At the moment, column statistics tables in the metastore schema look like this (it's similar for PART_COL_STATS):

      CREATE TABLE "APP"."TAB_COL_STATS"(
          "CAT_NAME" VARCHAR(256) NOT NULL,
          "DB_NAME" VARCHAR(128) NOT NULL,
          "TABLE_NAME" VARCHAR(256) NOT NULL,
          "COLUMN_NAME" VARCHAR(767) NOT NULL,
          "COLUMN_TYPE" VARCHAR(128) NOT NULL,
          "LONG_LOW_VALUE" BIGINT,
          "LONG_HIGH_VALUE" BIGINT,
          "DOUBLE_LOW_VALUE" DOUBLE,
          "DOUBLE_HIGH_VALUE" DOUBLE,
          "BIG_DECIMAL_LOW_VALUE" VARCHAR(4000),
          "BIG_DECIMAL_HIGH_VALUE" VARCHAR(4000),
          "NUM_DISTINCTS" BIGINT,
          "NUM_NULLS" BIGINT NOT NULL,
          "AVG_COL_LEN" DOUBLE,
          "MAX_COL_LEN" BIGINT,
          "NUM_TRUES" BIGINT,
          "NUM_FALSES" BIGINT,
          "LAST_ANALYZED" BIGINT,
          "CS_ID" BIGINT NOT NULL,
          "TBL_ID" BIGINT NOT NULL,
          "BIT_VECTOR" BLOB,
          "ENGINE" VARCHAR(128) NOT NULL
      );
      

      The idea is to have a single blob named STATISTICS to replace them, as follows:

      CREATE TABLE "APP"."TAB_COL_STATS"(
          "CAT_NAME" VARCHAR(256) NOT NULL,
          "DB_NAME" VARCHAR(128) NOT NULL,
          "TABLE_NAME" VARCHAR(256) NOT NULL,
          "COLUMN_NAME" VARCHAR(767) NOT NULL,
          "COLUMN_TYPE" VARCHAR(128) NOT NULL,
          "STATISTICS" BLOB,
          "LAST_ANALYZED" BIGINT,
          "CS_ID" BIGINT NOT NULL,
          "TBL_ID" BIGINT NOT NULL,
          "ENGINE" VARCHAR(128) NOT NULL
      );
      

      The STATISTICS column could be the serialization of a Json-encoded string, which will be consumed in a "schema-on-read" fashion.

      At first at least the removed column statistics will be encoded in the STATISTICS column, but since each "consumer" will read the portion of the schema it is interested into, multiple engines (see the ENGINE column) can read and write statistics as they deem fit.

      Another advantage is that, if we plan to add more statistics in the future, we won't need to change the thrift interface for the metastore again.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              asolimando Alessandro Solimando
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: