Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19256 Hive bucketing write support
  3. SPARK-17495

Hive hash implementation

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.2.0
    • 2.2.0
    • SQL
    • None

    Description

      Spark internally uses Murmur3Hash for partitioning. This is different from the one used by Hive. For queries which use bucketing this leads to different results if one tries the same query on both engines. For us, we want users to have backward compatibility to that one can switch parts of applications across the engines without observing regressions.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-31162 Provide Configuration Parameter to select/enforce the Hive Hash for Bucketing

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue SPARK-16904 Removal of Hive Built-in Hash Functions and TestHiveFunctionRegistry

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. HIVE-18910 Migrate to Murmur hash for shuffle and bucketing

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #15047 [Github] Pull Request #15047 (tejasapatil)

          Pull request #17049 [Github] Pull Request #17049 (tejasapatil)

          Pull request #17056 [Github] Pull Request #17056 (tejasapatil)

          Pull request #17062 [Github] Pull Request #17062 (tejasapatil)

          Activity

            People

              tejasp Tejas Patil
              tejasp Tejas Patil
              Votes:
              2 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: