[SPARK-49893] File data sources don't respect provided user schema nullability for all APIs

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Uros Stankovic
• Created: 07/Oct/24 13:03
• Updated: 07/Oct/24 13:03

Description

DataFrameReader has 3 APIs for JSON reading

• json(DataSet[String])
• json(Rdd)
• json(filePath)

First two APIs respects provided user schema nullability when spark flag `spark.sql.legacy.respectNullabilityInTextDatasetConversion` is set to true, but third one does not respect and provided schema nullability is always overriden to true.

Some users just want to have a validation of data and to get exception when some field is nullable.