[SPARK-49700] Switch interface and implementations

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Connect, SQL
• Labels: pull-request-available
• Epic Link: Unified SQL Scala Interface
• Assignee: Herman van Hövell
• Reporter: Herman van Hövell
• Created: 18/Sep/24 20:34
• Updated: 20/Nov/24 15:17

Description

Move the interfaces to `org.apache.spark.sql`.

Move implementation specific classes into the `classic` or `connect` packages.

Tests in sql/core will still run against classic.

Issue Links

• blocks: SPARK-49428 Rename/Reorg Connect Client Packages
• blocks: SPARK-49437 Create interface implementation tests
• blocks: SPARK-49283 Create a generic builder for SparkSession implementations
• is blocked by: SPARK-49710 Make sure internal/dev API is properly annotated in SparkSession
• is blocked by: SPARK-49282 Create shared interface for SparkSession.Builder
• is blocked by: SPARK-49286 Move the Avro and Protobuf functions.scala files to sql/api
• is blocked by: SPARK-49369 Create implicit Column conversions to reduce upgrade issues
• is blocked by: SPARK-49416 Create a shared interface for DataStreamReader
• is blocked by: SPARK-49417 Create a shared interface for StreamingQueryManager
• is blocked by: SPARK-49418 Active/default ThreadLocals for shared SparkSession
• is blocked by: SPARK-49434 Move org.apache.spark.sql.expressions.javalang.typed/scalalang.type to sql/api
• is blocked by: SPARK-49435 Move ReduceAggregator to sql/api
• is blocked by: SPARK-49569 Introduce Shim for missing spark/core classes
• is blocked by: SPARK-49712 Replace org.apache.spark.sql.encoderFor with AgnosticEncoders.agnosticEncoderFor
• is blocked by: SPARK-50102 Add shims for all classic only classes
• links to: GitHub Pull Request #48818