[SPARK-32711] Write Hive ORC/Parquet bucketed table with hive murmur3hash (for Hive 3)

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Cheng Su
• Created: 27/Aug/20 06:09
• Updated: 29/Aug/20 18:56

Description

Hive ORC/Parquet write code path is same as data source v1 code path (FileFormatWriter). This JIRA is to add the support to write Hive ORC/Parquet bucketed table with hive murmur3hash. The change is to custom `bucketIdExpression` to use hive murmur3hash when the table is Hive bucketed table, and the Hive version is 3.x.y.

 

This will allow us write Hive/Presto-compatible bucketed table for Hive 3.