-
Type:
New Feature
-
Status: Patch Available
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 1.2.1, 2.1.1
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
Merge configuration parameter, like hive.merge.size.per.task , decide the average file after merge stage.
But we found it only work for file format like Textfile/SequenceFile. With RC/ORC file format, it does not work.
For RC/ORC file format we found the file size after merge stage, depends on parameter like {{mapreduce.input.fileinputformat.split.maxsize}.
it is better to use hive.merge.size.per.task to decide the the average file size for RC/ORC fileformat, which results in unifying.
Root Cause is for RC/ORC file format, merge class is MergeFileTask instead of MapRedTask for Textfile/SequenceFile. And MergeFileTask just has not accept the configuration value in MergeFileWork, so the solution is passing it into MergeFileTask