Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18206

Merge of RC/ORC file should follow other fileformate which use merge configuration parameter

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2.1, 2.1.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Merge configuration parameter, like hive.merge.size.per.task , decide the average file after merge stage.

      But we found it only work for file format like Textfile/SequenceFile. With RC/ORC file format, it does not work.

      For RC/ORC file format we found the file size after merge stage, depends on parameter like {{mapreduce.input.fileinputformat.split.maxsize}.

      it is better to use hive.merge.size.per.task to decide the the average file size for RC/ORC fileformat, which results in unifying.

      Root Cause is for RC/ORC file format, merge class is MergeFileTask instead of MapRedTask for Textfile/SequenceFile. And MergeFileTask just has not accept the configuration value in MergeFileWork, so the solution is passing it into MergeFileTask

        Attachments

        1. HIVE-18206.1.patch
          1 kB
          Wang Haihua
        2. HIVE-18206.2.patch
          1 kB
          Wang Haihua

          Activity

            People

            • Assignee:
              wanghaihua Wang Haihua
              Reporter:
              wanghaihua Wang Haihua
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: