[SPARK-50146] Consolidate configurable schema verification of createDataFrame

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Connect, PySpark
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Xinrong Meng
• Created: 28/Oct/24 06:48
• Updated: 3 hours ago

Description

verifySchema parameter of createDataFrame decides whether to verify data types of every row against schema.

In Spark Classic

Now it only takes effect for with createDataFrame with

• regular Python instances

We propose to make it work with createDataFrame with

• pyarrow.Table
• pandas.DataFrame with Arrow optimization
• pandas.DataFrame without Arrow optimization

In Spark Connect

Now it does not take effect.

We propose to make it work with all inputs.

Issue Links

• links to: GitHub Pull Request #48677

Sub-Tasks

• Standardize verifySchema parameter of createDataFrame in Spark Classic: Open
• Deprecate "spark.sql.execution.pandas.convertToArrowArraySafely" configuration: Open
• Implement verifySchema parameter of createDataFrame in Spark Connect: Open