[HIVE-11160] Auto-gather column stats

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: Statistics
• Target Version/s: 3.0.0
• Assignee: Pengcheng Xiong
• Reporter: Pengcheng Xiong
• Created: 01/Jul/15 20:35
• Updated: 27/Feb/24 22:23

Description

Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using "Analyze" command. In this patch, the column stats will also be collected automatically. More specifically, INSERT OVERWRITE will automatically create new column stats. INSERT INTO will automatically merge new column stats with existing ones.

Issue Links

• is blocked by: HIVE-11224 AggregateStatsCache triggers java.util.ConcurrentModificationException under some conditions
• is blocked by: HIVE-13210 Revert changes in HIVE-12994 related to metastore
• is blocked by: HIVE-13260 ReduceSinkDeDuplication throws exception when pRS key is empty
• is blocked by: HIVE-13261 Can not compute column stats for partition when schema evolves
• is related to: HIVE-12656 Turn hive.compute.query.using.stats on by default
• relates to: HIVE-17406 UDAF throws IllegalArgumentException for a complex input when column stats is not provided
• relates to: HIVE-12763 Use bit vector to track NDV
• links to: RB

Sub-Tasks

• thrift change: Closed
• Auto-gather column stats - phase 1: Closed
• "Create table like" command should initialize the basic stats for the table: Closed
• GenMRFileSink1.java may refer to a wrong MR task in multi-insert case: Resolved
• Support auto gather column stats for columns with trailing white spaces: Resolved
• Support stats computation for column in QuotedIdentifier: Resolved
• With column stats, mergejoin.q throws NPE: Resolved
• Column pruner should continue to work when SEL has more than 1 child: Resolved
• Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData: Resolved
• Fix failing test columnstats_partlvl_invalid_values when autogather column stats is on: Resolved
• analyze table compute statistics fails due to presence of Infinity value in double column: Resolved
• Skip column stats when colStats is empty: Closed
• ColumnStats merge should consider the accuracy of the current stats: Resolved
• Set column stats default as true when creating new tables/partitions: Closed
• Merge stats task and column stats task into a single task: Closed
• hive.optimize.bucketingsorting should compare the schema before removing RS: Closed
• remove ColumnStatsDesc usage from columnstatsupdatetask: Closed
• retire ANALYZE TABLE ... PARTIALSCAN: Closed
• TableScanOperator might miss vectorization on flag: Closed
• Merging Statistics are promoted to COMPLETE (most of the time): Closed
• Fix exception on tables handled by HBaseHandler if columnsstats are auto-gathered: Closed
• Remove mixed partitions/table schema support: Resolved
• Aggregation of an empty set doesn't pass constants to the UDAF: Resolved
• Fix StatsUtils.combineRange to combine intervals: Closed
• Stats: create materialized view should also collect stats: Closed
• Stats: Remove usage of clone() methods: Closed
• Improve size estimation for array() to be not 0: Closed
• Fix columnstats problem in case schema evolution: Closed
• Enable auto-gather column stats by default: Closed
• Stats: rownum estimation from datasize underestimates in most cases: Closed
• Fix test TestAcidOnTez: Resolved
• in case basic stats are missing; rowcount estimation depends on the selected columns size: Resolved
• Remove hive.stats.atomic: Closed
• Columnstats gather on mm tables: re-enable disabled test: Closed
• Partitioned tables statistics can go wrong in basic stats mixed case: Resolved
• Fix columnstats merge NPE: Closed
• Fill stats for temporary tables: Resolved
• Aggregate row traffic for acid tables: Closed
• Deprecate HIVESTATSAUTOGATHER: Patch Available
• Move filesystem stats collection from metastore to ql: Patch Available
• StatsUtils.getColStatisticsFromExprMap may only provide info for a column once: Patch Available
• Deprecate hive.typecheck.on.insert: Open
• UpdateColumnStatsTask should set column stats as inaccurate: Open
• Support CTAS for auto gather column stats: Closed
• improve explain when invalidate stats: Open
• Consolidate basic stats logic for standalone table / partitioned: Open
• Support date type for merging column stats: Open
• Support vectorization for UDAF compute_stats: Open
• Stats: Consolidate stat state for limit 0 and where false: Open
• Tables which are known to be empty should not have NONE basic stat state: Open
• Possible misuse of getDataSizeFromColumnStats: Open
• Derby throws java.lang.StackOverflowError when it tries to get column stats from a table with thousands columns: Open
• Revise basic stat states for estimations: Open
• Investigate bucketed table stats: Open
• Differentiate table level stat / operator level stats: Open
• Column stats are not autogathered for materialized views: Closed
• Make StatsTask use less metastore calls: Open
• Incorrect rownum estimation in joins: Open
• Estimate avgrowsize for stats calc in mixed case: Open
• Statistics: rawDataSize seems to be underestimated for text tables: Open
• Support "analyze table T": In Progress
• Support date type for column stats autogather: Closed
• Remove column statistics collection task from merge statement plan: Closed