[HIVE-20079] Populate more accurate rawDataSize for parquet format

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: File Formats
• Assignee: Aihua Xu
• Reporter: Aihua Xu
• Created: 03/Jul/18 23:36
• Updated: 17/Nov/22 08:55
• Resolved: 26/Feb/19 12:48

Description

Run the following queries and you will see the raw data for the table is 4 (that is the number of fields) incorrectly. We need to populate correct data size so data can be split properly.

SET hive.stats.autogather=true;
CREATE TABLE parquet_stats (id int,str string) STORED AS PARQUET;
INSERT INTO parquet_stats values(0, 'this is string 0'), (1, 'string 1');
DESC FORMATTED parquet_stats;

Table Parameters:
	COLUMN_STATS_ACCURATE	true
	numFiles            	1
	numRows             	2
	rawDataSize         	4
	totalSize           	373
	transient_lastDdlTime	1530660523

Issue Links

• is duplicated by: HIVE-20523 Improve table statistics for Parquet format
• is related to: HIVE-21284 StatsWork should use footer scan for Parquet
• is related to: HIVE-16887 Parquet rawDataSize Under Reported