[HIVE-19326] stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Statistics
• Target Version/s: 3.1.0
• Assignee: Zoltan Haindrich
• Reporter: Sergey Shelukhin
• Created: 26/Apr/18 20:18
• Updated: 17/Nov/22 08:54
• Resolved: 03/Jul/18 17:27

Description

Found when investigating the results change after converting tables to MM, turns out the MM result is correct but the current one is not.
The test ends like so:

desc formatted small_alltypesorc_a;
ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
desc formatted small_alltypesorc_a;
insert into table small_alltypesorc_a select * from small_alltypesorc1a;
desc formatted small_alltypesorc_a;

The results from the descs in the golden file are:

	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\"}
	numFiles            	1                   
	numRows             	5                               
...
	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\"}
	numFiles            	1                   
	numRows             	15                                
...
	COLUMN_STATS_ACCURATE	{\"BASIC_STATS\":\"true\"}
	numFiles            	2                   
	numRows             	20                              

Note the result change after analyze - the original nomRows is inaccurate, but BASIC_STATS is set to true.

I am assuming with metadata only optimization this can produce incorrect results.

Issue Links

• contains: HIVE-19543 Enable TestMiniLlapLocalCliDriver#union_fast_stats.q
• is blocked by: HIVE-19237 Only use an operatorId once in a plan
• relates to: HIVE-20024 Determine the source of data size fluctuation in union_fast_stats
• links to: rb