[SPARK-39849] Dataset.as(StructType) fills missing new columns with null value

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 24/Jul/22 09:19
• Updated: 29/Jul/22 12:09
• Resolved: 29/Jul/22 12:09

Description

As a followup of https://github.com/apache/spark/pull/37011#discussion_r917700960 , it would be great to fill missing new columns with null values, instead of failing out loud. Note it would only work for nullable columns.

Issue Links

• links to: [Github] Pull Request #37264 (c21)
• links to: [Github] Pull Request #37264 (c21)