[SPARK-36758] SQL column nullable setting not retained as part of spark read

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: PySpark

Environment

Databricks:

Runtime 8.2

Spark 3.1.1

Language

English

• Assignee: Unassigned
• Reporter: Darren Price
• Created: 14/Sep/21 20:52
• Updated: 15/Sep/21 01:15

Description

When reading in a column set as not null this is not retained as part of the spark.read.
All columns are showing as nullable = true

Is there a way to change this behaviour to retain the null setting from the source?

See here for more info

https://github.com/microsoft/sql-spark-connector/issues/121

 

Example code from databricks:

tableName = "dbo.MyTable"

df = spark.read
.format("com.microsoft.sqlserver.jdbc.spark")
.option("url", myJdbcUrl)
.option("accessToken", accessToken)
.option("dbTable", tableName)
.load()

df.printSchema()