[HIVE-24224] Fix skipping header/footer for Hive on Tez on compressed files

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: Panagiotis Garefalakis
• Reporter: Panagiotis Garefalakis
• Created: 02/Oct/20 11:14
• Updated: 17/Nov/22 08:48
• Resolved: 06/Oct/20 04:13

Description

Compressed file with Hive on Tez returns header and footers - for both select * and select count ( * ):

printf "offset,id,other\n9,\"20200315 X00 1356\",123\n17,\"20200315 X00 1357\",123\nrst,rst,rst" > data.csv
hdfs dfs -put -f data.csv /apps/hive/warehouse/bz2test/bz2tbl1/
bzip2 -f data.csv 
hdfs dfs -put -f data.csv.bz2 /apps/hive/warehouse/bz2test/bz2tbl2/

beeline -e "CREATE EXTERNAL TABLE default.bz2tst2 (
  sequence   int,
  id         string,
  other      string) 
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
LOCATION '/apps/hive/warehouse/bz2test/bz2tbl2' 
TBLPROPERTIES (
  'skip.header.line.count'='1',
  'skip.footer.line.count'='1');"

beeline -e "
  SET hive.fetch.task.conversion = none;
  SELECT * FROM default.bz2tst2;"
+-------------------+--------------------+----------------+
| bz2tst2.sequence  |     bz2tst2.id     | bz2tst2.other  |
+-------------------+--------------------+----------------+
| offset            | id                 | other          |
| 9                 | 20200315 X00 1356  | 123            |
| 17                | 20200315 X00 1357  | 123            |
| rst               | rst                | rst            |
+-------------------+--------------------+----------------+

PS: HIVE-22769 addressed the issue for Hive on LLAP.

Issue Links

• fixes: HIVE-21924 Split text files even if header/footer exists
• is related to: HIVE-22769 Incorrect query results and query failure during split generation for compressed text files
• relates to: HIVE-24381 Compressed text input returns 0 rows if skip header/footer is mentioned
• links to: GitHub Pull Request #1546