[SPARK-48883] In spark ML, replace RDD read / write API invocation with Dataframe read / write API

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: ML
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Weichen Xu
• Created: 12/Jul/24 12:39
• Updated: 16/Jul/24 23:56
• Resolved: 16/Jul/24 23:56

Description

In spark ML, replace RDD read / write API invocation with Dataframe read / write API.

 

Motivation:

In databricks runtime, RDD read / write API has some issue for certain storage types that requires the account key, but Dataframe read / write API works.

Issue Links

• links to: GitHub Pull Request #47328
• links to: GitHub Pull Request #47341