[HIVE-25505] Incorrect results with header. skip.header.line.count if first line is blank

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: HiveServer2
• Labels: pull-request-available
• Assignee: Panagiotis Garefalakis
• Reporter: Steve Carlin
• Created: 07/Sep/21 16:16
• Updated: 17/Nov/22 08:48
• Resolved: 20/Oct/21 11:38

Description

aAtable with header. skip.header.line.count=1 does not skip the first line if it is blank, except in a fetch task.

To reproduce, create a csv table, ans set header. skip.header.line.count=1 in table properties.

In the table location, create a single file, with a blank (empty) first line, and say 2 further lines.

If you do a select * on it, you see 2 rows (correct)
If you do select count on it, you get 3 (incorrect)

CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
  LOCATION '${system:test.tmp.dir}/testcase1'
  TBLPROPERTIES ("skip.header.line.count"="1");

SET hive.fetch.task.conversion = more;
select * from testcase1;
select count(*) from testcase1;


set hive.fetch.task.conversion=none;
select * from testcase1;
select count(*) from testcase1;

Test file:

1,2019-12-31
2,2019-12-31
3,2019-12-31



Should both yield (with the above test file):
#### A masked pattern was here ####
1	2019-12-31
2	2019-12-31
3	2019-12-31

3

Issue Links

• relates to: HIVE-19943 Header values keep showing up in result sets
• relates to: HIVE-22769 Incorrect query results and query failure during split generation for compressed text files
• links to: GitHub Pull Request #2717