[SPARK-45784] Introduce clustering mechanism to Spark

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Terry Kim
• Created: 03/Nov/23 23:01
• Updated: 03/Nov/23 23:03

Description

This proposes to introduce a clustering mechanism such that different data sources (e.g., Delta, Iceberg, etc.) can implement format specific clustering.

Sub-Tasks

• Introduce CLUSTER BY SQL clause to CREATE/REPLACE TABLE: Resolved
• Support Catalog.listColumns() for clustering columns: Resolved
• Support SHOW CREATE TABLE for clustering columns: Open
• Support DESCRIBE TABLE for clustering columns: Resolved
• Introduce ALTER TABLE ... CLUSTER BY to change clustering columns: Resolved
• Add clusterBy DataFrameWriter API for Scala: Resolved
• Add clusterBy DataFrameWriter API for Python: Resolved
• Add clusterBy DataStreamWriter API for Scala: Resolved
• Add clusterBy DataStreamWriter API for Python: Resolved