Hive
  1. Hive
  2. HIVE-1181

Make it optional to check file format

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Until MAPREDUCE-1504 is fixed, we need to introduce a way to avoid leaking file descriptor because of file format check in the load command.

      We already have the option in MoveTask. We just need to add a HiveConf var to make it configurable.

      1. hive-1181-trunk-4.patch
        5 kB
        He Yongqiang
      2. hive-1181-trunk-3.patch
        3 kB
        He Yongqiang
      3. hive-1181-trunk-1.patch
        3 kB
        He Yongqiang
      4. hive-1181-branch0.5-3.patch
        5 kB
        He Yongqiang
      5. hive-1181-branch0.5-2.patch
        5 kB
        He Yongqiang
      6. hive-1181-branch0.5-1.patch
        3 kB
        He Yongqiang

        Activity

        Hide
        He Yongqiang added a comment -

        This is 0.5 specific issue, or we also need to make it for trunk?

        Show
        He Yongqiang added a comment - This is 0.5 specific issue, or we also need to make it for trunk?
        Hide
        Zheng Shao added a comment -

        We want to make it available for trunk also.

        Let's give the parameter some more detailed descriptions. The current description is not useful.
        Something like "Whether to check file format or not when loading data files" is much better.

        Show
        Zheng Shao added a comment - We want to make it available for trunk also. Let's give the parameter some more detailed descriptions. The current description is not useful. Something like "Whether to check file format or not when loading data files" is much better.
        Hide
        Zheng Shao added a comment -

        Talked with Yongqiang offline. We need to add a test to make sure the option can actually avoid the file format check.

        Show
        Zheng Shao added a comment - Talked with Yongqiang offline. We need to add a test to make sure the option can actually avoid the file format check.
        Hide
        Zheng Shao added a comment -

        The "drop table" command is dropping wrong tables. Also we should have a newline at the end of the file.

        --- ql/src/test/queries/clientpositive/disable_file_format_check.q	(revision 0)
        +++ ql/src/test/queries/clientpositive/disable_file_format_check.q	(revision 0)
        @@ -0,0 +1,8 @@
        +set hive.fileformat.check = false;
        +create table kv_fileformat_check_txt (key string, value string) stored as textfile;
        +load data local inpath '../data/files/kv1.seq' overwrite into table kv_fileformat_check_txt;
        +
        +create table kv_fileformat_check_seq (key string, value string) stored as sequencefile;
        +load data local inpath '../data/files/kv1.txt' overwrite into table kv_fileformat_check_seq;
        +
        +drop table kv_fileformat_check;
        \ No newline at end of file
        
        Show
        Zheng Shao added a comment - The "drop table" command is dropping wrong tables. Also we should have a newline at the end of the file. --- ql/src/test/queries/clientpositive/disable_file_format_check.q (revision 0) +++ ql/src/test/queries/clientpositive/disable_file_format_check.q (revision 0) @@ -0,0 +1,8 @@ +set hive.fileformat.check = false ; +create table kv_fileformat_check_txt (key string, value string) stored as textfile; +load data local inpath '../data/files/kv1.seq' overwrite into table kv_fileformat_check_txt; + +create table kv_fileformat_check_seq (key string, value string) stored as sequencefile; +load data local inpath '../data/files/kv1.txt' overwrite into table kv_fileformat_check_seq; + +drop table kv_fileformat_check; \ No newline at end of file
        Hide
        Zheng Shao added a comment -

        Committed. Thanks Yongqiang.

        Show
        Zheng Shao added a comment - Committed. Thanks Yongqiang.

          People

          • Assignee:
            He Yongqiang
            Reporter:
            Zheng Shao
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development