[HIVE-23831] Make auto reducer parallelism min threshold configurable.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Component/s: Hive
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Shohei Okumiya
• Reporter: Shohei Okumiya
• Created: 10/Jul/20 05:14
• Updated: 15/Aug/23 06:33
• Resolved: 26/Jun/23 05:49

Description

Hive on Tez disables auto-reduce and uses the maximum number of partitions when the minimum number of reducers, `estimated number * min.partition.factor`, is less than or equal to 1.

https://github.com/apache/hive/blob/f98e136bdd5642e3de10d2fd1a4c14d1d6762113/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L121-L131

https://issues.apache.org/jira/browse/HIVE-14200

 

This can make Tez lose chance to optimize # of reducers especially in an environment with small `hive.tez.min.partition.factor` or big `hive.tez.max.partition.factor`.

Issue Links

• relates to: HIVE-14200 Tez: disable auto-reducer parallelism when reducer-count * min.partition.factor < 1.0
• links to: GitHub Pull Request #4432