[SPARK-41286] Build, package and infrastructure for Spark Connect

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 3.4.0
• Component/s: Connect
• Epic Link: Spark Connect
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Shepherd: Hyukjin Kwon
• Created: 28/Nov/22 02:26
• Updated: 05/Jan/23 00:25
• Resolved: 05/Jan/23 00:25

Sub-Tasks

• Check the generated python protos in GitHub Actions: Resolved
• Enforce Scalafmt for Spark Connect Module: Resolved
• pin 'grpcio==1.48.1' 'protobuf==4.21.6': Resolved
• Homogenize the protobuf version across server and client: Resolved
• protoc-3.21.9-linux-x86_64.exe requires GLIBC_2.14: Resolved
• Upgrade buf to v1.9.0: Resolved
• Check and upgrade buf.build/protocolbuffers/plugins/python to 3.19.5: Resolved
• Add `CONNECT` label: Resolved
• protoc-3.21.1-linux-x86_64.exe requires GLIBC_2.14: Resolved
• Update sbt-protoc to 1.0.6: Resolved
• Avoid embedding Spark Connect in the Apache Spark binary release: Resolved
• Shade more dependency to be able to run separately: Resolved
• Avoid hardcoded versions in SBT build: Resolved
• Replace shaded netty with grpc netty to avoid double shaded dependency.: Resolved
• Re-generate Spark Connect Python protos: Resolved
• Sharing namespace between PySpark and Spark Connect: Resolved
• Add Spark Connect and protobuf into setup.py with specifying dependencies: Resolved
• Upgrade buf to v1.11.0: Resolved
• Support PyPI packaging without JVM: Resolved
• Distribution build for Spark Connect does not work with Spark Shell: Resolved
• Upgrade buf to v1.15.0: Resolved
• Upgrade buf to v1.15.1: Resolved