[HIVE-22495] Parquet count(*) read in all data

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: Reader
• Assignee: Jason Xu
• Reporter: Jason Xu
• Created: 14/Nov/19 13:19
• Updated: 01/Oct/21 11:37

Description

Running a hive query on a Parquet table

select count ( * ) from test_table

The query read in all data (all columns) instead of just metadata.

For comparison, hive 0.13 and Spark read in much less data with my test table.

 

engine	HDFS data read
Hive 2.3.4	452.9 MB
Hive 0.13	22.5 KB
Spark	41.6 KB

 

Seems cause is that Parquet read support fall back to file schema if indexColumnsWanted is empty, logic still exist in master branch.

Don't know why this empty list check was added, please suggest if there're any other impact.