[HIVE-3733] Improve Hive's logic for conditional merge

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Zhenxiao Luo
• Reporter: Pradeep Kamath
• Created: 22/Nov/12 01:17
• Updated: 31/Dec/12 04:48

Description

If the config hive.merge.mapfiles is set to true and hive.merge.mapredfiles is set to false then when hive encounters a FileSinkOperator when generating map reduce tasks, it will look at the entire job to see if it has a reducer, if it does it will not merge. Instead it should be check if the FileSinkOperator is a child of the reducer. This means that outputs generated in the mapper will be merged, and outputs generated in the reducer will not be, the intended effect of setting those configs.

Simple repro:

set hive.merge.mapfiles=true;
set hive.merge.mapredfiles=false;
EXPLAIN
FROM <input_table>
INSERT OVERWRITE TABLE <output_table1> SELECT key, COUNT group by key
INSERT OVERWRITE TABLE <output_table2> SELECT *;

The output should contain a Conditional Operator, Mapred Stages, and Move tasks