[SPARK-36894] RDD.toDF should be synchronized with dispatched variants of SparkSession.createDataFrame

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: PySpark, SQL
• Assignee: Maciej Szymkiewicz
• Reporter: Maciej Szymkiewicz
• Created: 29/Sep/21 22:21
• Updated: 06/Jan/22 19:40
• Resolved: 03/Nov/21 12:11

Description

There are some variants that are supported:

• Providing a schema as a str object for RDD[RowLike] objects
• Providing a schema as a Tuple[str, ...] names
• Calling toDF on RDD of atomic values, when schema of str or AtomicType is provided.

Issue Links

• links to: [Github] Pull Request #34146 (zero323)
• links to: [Github] Pull Request #34146 (zero323)
• links to: [Github] Pull Request #34477 (sarutak)
• links to: [Github] Pull Request #34477 (sarutak)
• links to: [Github] Pull Request #34478 (zero323)
• links to: [Github] Pull Request #34478 (zero323)