[SPARK-42471] Distributed ML <> spark connect

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Connect, ML
• Epic Link: Spark Connect
• Assignee: Weichen Xu
• Reporter: Ruifeng Zheng
• Created: 17/Feb/23 05:18
• Updated: 15/Nov/23 10:22

Sub-Tasks

• Make spark connect supporting canceling job group: Open
• High level design doc for Distributed ML <> spark connect: Resolved
• Initial prototype implementation for PySparkML: Resolved
• Extract the common .ml classes to `mllib-common`: Resolved
• Make LiteralExpression support array: Resolved
• Factor literal value conversion out to connect-common: Resolved
• Helper function to convert proto literal to value in Python Client: Resolved
• Implement ml function {array_to_vector, vector_to_array}: Resolved
• Move `toCatalystValue` to connect-common: Resolved
• Make Torch Distributor compatible with Spark Connect: Resolved
• Torch Distributor support Local Mode: Resolved
• Add torch distributor data loader that loads data from spark partition data: Resolved
• Implement pyspark ML logistic regression estimator on top of torch distributor: Resolved
• Basic estimator / transformer / model / evaluator interfaces and basic transformer / evaluator implementation: Resolved
• Add spark DataFrame binary file format writer: Resolved
• Add API `copyLocalFileToHadoopFS`: Resolved
• Basic saving / loading implementation: Resolved
• Implement pipeline estimator: Resolved
• Implement cross validator estimator: Resolved
• Move namespace from `pyspark.mlv2` to `pyspark.ml.connect`: Resolved
• Implement classification evaluator: Resolved
• Add example code: Resolved
• Add pyspark "ml-connect" extras dependencies: Open
• Avoid Spark connect ML model to change input pandas dataframe: Resolved
• Add doc entry for `pyspark.ml.connect` module: Resolved
• Add vector assembler feature transformer: Resolved