[HIVE-5900] Bucketing and dynamic partitions should allow additional reducers

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Charles Pritchard
• Created: 27/Nov/13 02:57
• Updated: 27/Nov/13 02:57

Description

When a target table is clustered into buckets, hive.enforce.bucketing will limit the number of reducers to match the number of buckets. It should take into account the number of partitions * number of buckets.

SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.enforce.bucketing = true;
SET hive.enforce.sorting = true;
INSERT INTO TABLE dest PARTITION(part) SELECT * FROM origin;

In this case, when dest has 5 buckets, the number of reducers will be 5 – if there are 10 partitions, the maximum reducers should be 50, not 5.