Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1740

Impala cannot deal with skip.header.line.count

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.1.1
    • Fix Version/s: Impala 2.6.0
    • Component/s: Backend
    • Labels:

      Description

      HIVE-5795 introduces parameter skip.header.line.count.
      But impala cannot deal with the parameter for now.

      Here's my testcase.

      sudo -u hive hive shell
      create table t1 (c1 bigint, c2 double)
      row format delimited fields terminated by ',' lines terminated by '\n'
      tblproperties("skip.header.line.count"="1");
      exit;

      cat <<EOF >/tmp/test.txt
      num1,num2
      1,1
      2,2
      3,3
      EOF
      sudo -u impala hdfs dfs -put /tmp/test.txt /user/hive/warehouse/t1

      sudo -u hive hive shell
      select * from t1;
      exit;

      sudo -u impala impala-shell
      invalidate metadata;
      select * from t1;

      result of hive(skip first string row)

      hive> select * from t1;
      OK
      1 1.0
      2 2.0
      3 3.0
      Time taken: 1.681 seconds, Fetched: 3 row(s)

      result of impala (cannot skip first row and Warning printed)

      [n2.lab.cloudera.com:21000] > select * from t1;
      Query: select * from t1
      ----------+
      c1 c2

      ----------+
      NULL NULL
      1 1
      2 2
      3 3

      ----------+
      WARNINGS: Backend 0:Error converting column: 0 TO BIGINT (Data is: num1)
      Error converting column: 1 TO DOUBLE (Data is: num2)
      file: hdfs://n1.lab.cloudera.com:8020/user/hive/warehouse/t1/test.txt
      record: num1,num2

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lv Lars Volker
                Reporter:
                tomo Tomoaki Yano
              • Votes:
                6 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: