[HIVE-26313] Aggregate all column statistics into a single field in metastore

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0-alpha-2
• Component/s: Standalone Metastore, Statistics
• Labels: backward-incompatible
• Assignee: Unassigned Assign to me
• Reporter: Alessandro Solimando
• Created: 10/Jun/22 13:50
• Updated: 20/Mar/23 15:19

Description

At the moment, column statistics tables in the metastore schema look like this (it's similar for PART_COL_STATS):

CREATE TABLE "APP"."TAB_COL_STATS"(
    "CAT_NAME" VARCHAR(256) NOT NULL,
    "DB_NAME" VARCHAR(128) NOT NULL,
    "TABLE_NAME" VARCHAR(256) NOT NULL,
    "COLUMN_NAME" VARCHAR(767) NOT NULL,
    "COLUMN_TYPE" VARCHAR(128) NOT NULL,
    "LONG_LOW_VALUE" BIGINT,
    "LONG_HIGH_VALUE" BIGINT,
    "DOUBLE_LOW_VALUE" DOUBLE,
    "DOUBLE_HIGH_VALUE" DOUBLE,
    "BIG_DECIMAL_LOW_VALUE" VARCHAR(4000),
    "BIG_DECIMAL_HIGH_VALUE" VARCHAR(4000),
    "NUM_DISTINCTS" BIGINT,
    "NUM_NULLS" BIGINT NOT NULL,
    "AVG_COL_LEN" DOUBLE,
    "MAX_COL_LEN" BIGINT,
    "NUM_TRUES" BIGINT,
    "NUM_FALSES" BIGINT,
    "LAST_ANALYZED" BIGINT,
    "CS_ID" BIGINT NOT NULL,
    "TBL_ID" BIGINT NOT NULL,
    "BIT_VECTOR" BLOB,
    "ENGINE" VARCHAR(128) NOT NULL
);

The idea is to have a single blob named STATISTICS to replace them, as follows:

CREATE TABLE "APP"."TAB_COL_STATS"(
    "CAT_NAME" VARCHAR(256) NOT NULL,
    "DB_NAME" VARCHAR(128) NOT NULL,
    "TABLE_NAME" VARCHAR(256) NOT NULL,
    "COLUMN_NAME" VARCHAR(767) NOT NULL,
    "COLUMN_TYPE" VARCHAR(128) NOT NULL,
    "STATISTICS" BLOB,
    "LAST_ANALYZED" BIGINT,
    "CS_ID" BIGINT NOT NULL,
    "TBL_ID" BIGINT NOT NULL,
    "ENGINE" VARCHAR(128) NOT NULL
);

The STATISTICS column could be the serialization of a Json-encoded string, which will be consumed in a "schema-on-read" fashion.

At first at least the removed column statistics will be encoded in the STATISTICS column, but since each "consumer" will read the portion of the schema it is interested into, multiple engines (see the ENGINE column) can read and write statistics as they deem fit.

Another advantage is that, if we plan to add more statistics in the future, we won't need to change the thrift interface for the metastore again.

Issue Links

• relates to: HIVE-26221 Add histogram-based column statistics