[SPARK-41589] PyTorch Distributor

• Type: Umbrella
• Status: Reopened
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: ML, PySpark
• Assignee: Unassigned
• Reporter: Rithwik Ediga Lakhamsani
• Shepherd: Hyukjin Kwon
• Created: 20/Dec/22 00:14
• Updated: 18/Jan/23 02:30

Description

This is a project to make it easier for PySpark users to distribute PyTorch code using PySpark. The corresponding Design Document can give more context. This was a project determined by the Databricks ML Training Team; please reach out to Hyukjin Kwon (Spark-side) or Rithwik Ediga Lakhamsani for more context.

Issue Links

• links to: [Github] Pull Request #39146 (rithwik-db)
• links to: [Github] Pull Request #39490 (HyukjinKwon)

Sub-Tasks

• Address General Fixes: Open