[HIVE-11160] Auto-gather column stats

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: Statistics
• Target Version/s: 3.0.0
• Assignee: Pengcheng Xiong
• Reporter: Pengcheng Xiong
• Created: 01/Jul/15 20:35
• Updated: 27/Feb/24 22:23

Description

Hive will collect table stats when set hive.stats.autogather=true during the INSERT OVERWRITE command. And then the users need to collect the column stats themselves using "Analyze" command. In this patch, the column stats will also be collected automatically. More specifically, INSERT OVERWRITE will automatically create new column stats. INSERT INTO will automatically merge new column stats with existing ones.

Issue Links

• is blocked by: HIVE-11224 AggregateStatsCache triggers java.util.ConcurrentModificationException under some conditions
• is blocked by: HIVE-13210 Revert changes in HIVE-12994 related to metastore
• is blocked by: HIVE-13260 ReduceSinkDeDuplication throws exception when pRS key is empty
• is blocked by: HIVE-13261 Can not compute column stats for partition when schema evolves
• is related to: HIVE-12656 Turn hive.compute.query.using.stats on by default
• relates to: HIVE-17406 UDAF throws IllegalArgumentException for a complex input when column stats is not provided
• relates to: HIVE-12763 Use bit vector to track NDV
• links to: RB

Sub-Tasks

• Deprecate HIVESTATSAUTOGATHER: Patch Available
• Move filesystem stats collection from metastore to ql: Patch Available
• StatsUtils.getColStatisticsFromExprMap may only provide info for a column once: Patch Available
• Deprecate hive.typecheck.on.insert: Open
• UpdateColumnStatsTask should set column stats as inaccurate: Open
• improve explain when invalidate stats: Open
• Consolidate basic stats logic for standalone table / partitioned: Open
• Support date type for merging column stats: Open
• Support vectorization for UDAF compute_stats: Open
• Stats: Consolidate stat state for limit 0 and where false: Open
• Tables which are known to be empty should not have NONE basic stat state: Open
• Possible misuse of getDataSizeFromColumnStats: Open
• Derby throws java.lang.StackOverflowError when it tries to get column stats from a table with thousands columns: Open
• Revise basic stat states for estimations: Open
• Investigate bucketed table stats: Open
• Differentiate table level stat / operator level stats: Open
• Make StatsTask use less metastore calls: Open
• Incorrect rownum estimation in joins: Open
• Estimate avgrowsize for stats calc in mixed case: Open
• Statistics: rawDataSize seems to be underestimated for text tables: Open
• Support "analyze table T": In Progress