[HIVE-7074] The reducer parallelism should be a prime number for better stride protection

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: Statistics
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 16/May/14 03:53
• Updated: 10/Jun/14 02:44
• Resolved: 10/Jun/14 02:44

Description

The current hive reducer parallelism results in stride issues with key distribution.

a JOIN generating even numbers will get strided onto only some of the reducers.

The probability of distribution skew is controlled by the number of common factors shared by the hashcode of the key and the number of buckets.

Using a prime number within the reducer estimation will cut that probability down by a significant amount.

Issue Links

• is superceded by: HIVE-7121 Use murmur hash to distribute HiveKey
• relates to: HIVE-7121 Use murmur hash to distribute HiveKey