[HIVE-18206] Merge of RC/ORC file should follow other fileformate which use merge configuration parameter - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Add vote

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Patch Available
Priority: Major
Resolution: Unresolved
Affects Version/s: 1.2.1, 2.1.1
Fix Version/s: None
Component/s: None
Labels:
None

Description

Merge configuration parameter, like hive.merge.size.per.task , decide the average file after merge stage.

But we found it only work for file format like Textfile/SequenceFile. With RC/ORC file format, it does not work.

For RC/ORC file format we found the file size after merge stage, depends on parameter like {{mapreduce.input.fileinputformat.split.maxsize}.

it is better to use hive.merge.size.per.task to decide the the average file size for RC/ORC fileformat, which results in unifying.

Root Cause is for RC/ORC file format, merge class is MergeFileTask instead of MapRedTask for Textfile/SequenceFile. And MergeFileTask just has not accept the configuration value in MergeFileWork, so the solution is passing it into MergeFileTask