[HIVE-28266] Iceberg: select count(*) from data_files metadata tables gives wrong result

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.1.0, 4.0.1
• Labels: hive-4.0.1-merged hive-4.0.1-must pull-request-available
• Assignee: Dmitriy Fingerman
• Reporter: Dmitriy Fingerman
• Created: 16/May/24 21:16
• Updated: 24/Sep/24 02:15
• Resolved: 22/May/24 08:12

Description

In Hive Iceberg, every table has a corresponding metadata table "*.data_files" that contains info about the files that contain table's data.

select count from a data_file metadata table returns number of rows in the data table instead of number of data files from the metadata table.

 

CREATE TABLE x (name VARCHAR(50), age TINYINT, num_clicks BIGINT) stored by iceberg stored as orc TBLPROPERTIES ('external.table.purge'='true','format-version'='2');
insert into x values 
('amy', 35, 123412344),
('adxfvy', 36, 123412534),
('amsdfyy', 37, 123417234),
('asafmy', 38, 123412534);
insert into x values 
('amerqwy', 39, 123441234),
('amyxzcv', 40, 123341234),
('erweramy', 45, 122341234);
Select * from default.x.data_files;
– Returns 2 records in the output
Select count from default.x.data_files;
– Returns 7 instead of 2

 

Issue Links

• links to: GitHub Pull Request #5253