[IMPALA-1740] Impala cannot deal with skip.header.line.count

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.1.1
• Fix Version/s: Impala 2.6.0
• Component/s: Backend
• Labels: usability
• Target Version: Product Backlog
• Assignee: Lars Volker
• Reporter: Tomoaki Yano
• Created: 06/Feb/15 07:38
• Updated: 21/Jun/18 22:36
• Resolved: 24/May/16 09:57

Description

HIVE-5795 introduces parameter skip.header.line.count.
But impala cannot deal with the parameter for now.

Here's my testcase.
–
sudo -u hive hive shell
create table t1 (c1 bigint, c2 double)
row format delimited fields terminated by ',' lines terminated by '\n'
tblproperties("skip.header.line.count"="1");
exit;

cat <<EOF >/tmp/test.txt
num1,num2
1,1
2,2
3,3
EOF
sudo -u impala hdfs dfs -put /tmp/test.txt /user/hive/warehouse/t1

sudo -u hive hive shell
select * from t1;
exit;

sudo -u impala impala-shell
invalidate metadata;
select * from t1;
–

result of hive(skip first string row)
–
hive> select * from t1;
OK
1 1.0
2 2.0
3 3.0
Time taken: 1.681 seconds, Fetched: 3 row(s)
–

result of impala (cannot skip first row and Warning printed)
–
[n2.lab.cloudera.com:21000] > select * from t1;
Query: select * from t1
----------+
c1 c2

----------+
NULL NULL
1 1
2 2
3 3

----------+
WARNINGS: Backend 0:Error converting column: 0 TO BIGINT (Data is: num1)
Error converting column: 1 TO DOUBLE (Data is: num2)
file: hdfs://n1.lab.cloudera.com:8020/user/hive/warehouse/t1/test.txt
record: num1,num2
–

Issue Links

• is duplicated by: IMPALA-2453 Support skip row when creating external table
• is related to: IMPALA-7196 Impala is not avoiding footers even after mentioning 'skip.footer.line.count' in ddl
• relates to: IMPALA-3458 Test failing: insert not supported for compressed text