[SPARK-17495] Hive hash implementation

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Tejas Patil
• Reporter: Tejas Patil
• Created: 10/Sep/16 19:33
• Updated: 12/Dec/22 18:10
• Resolved: 16/Mar/20 01:09

Description

Spark internally uses Murmur3Hash for partitioning. This is different from the one used by Hive. For queries which use bucketing this leads to different results if one tries the same query on both engines. For us, we want users to have backward compatibility to that one can switch parts of applications across the engines without observing regressions.

Issue Links

• is related to: SPARK-31162 Provide Configuration Parameter to select/enforce the Hive Hash for Bucketing
• is related to: SPARK-16904 Removal of Hive Built-in Hash Functions and TestHiveFunctionRegistry
• is related to: HIVE-18910 Migrate to Murmur hash for shuffle and bucketing
• links to: [Github] Pull Request #15047 (tejasapatil)
• links to: [Github] Pull Request #17049 (tejasapatil)
• links to: [Github] Pull Request #17056 (tejasapatil)
• links to: [Github] Pull Request #17062 (tejasapatil)