[HIVE-18206] Merge of RC/ORC file should follow other fileformate which use merge configuration parameter

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.1, 2.1.1
• Assignee: Wang Haihua
• Reporter: Wang Haihua
• Created: 03/Dec/17 15:09
• Updated: 08/Dec/17 06:50

Description

Merge configuration parameter, like hive.merge.size.per.task , decide the average file after merge stage.

But we found it only work for file format like Textfile/SequenceFile. With RC/ORC file format, it does not work.

For RC/ORC file format we found the file size after merge stage, depends on parameter like {{mapreduce.input.fileinputformat.split.maxsize}.

it is better to use hive.merge.size.per.task to decide the the average file size for RC/ORC fileformat, which results in unifying.

Root Cause is for RC/ORC file format, merge class is MergeFileTask instead of MapRedTask for Textfile/SequenceFile. And MergeFileTask just has not accept the configuration value in MergeFileWork, so the solution is passing it into MergeFileTask