[SPARK-27233] Schema of ArrayType change after saveAsTable and read

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.0
• Component/s: SQL

Environment

Spark 2.4

Scala 2.11

• Assignee: Unassigned
• Reporter: Kritsada Limpawatkul
• Created: 21/Mar/19 17:04
• Updated: 12/Dec/22 18:10
• Resolved: 25/Mar/19 01:35

Description

This is code for reproducing.

val testTable = "testseq"
val df = Seq(Some(Seq(1L,2L,3L)), None).toDF("seq")
sqlContext.sql(s"DROP TABLE IF EXISTS $testTable")
df.write.format("parquet").saveAsTable(testTable)
val res = sqlContext.table(testTable)
assert(df.schema === res.schema)

My code is trying to save dataframe with array type and read from that table. After checking schema, I found that schema has change.

This is the assertion message.

Expected :StructType(StructField(seq,ArrayType(LongType,true),true))
Actual :StructType(StructField(seq,ArrayType(LongType,false),true))

containsNull in ArrayType changes from false to true after reading from the table.

Issue Links

• duplicates: SPARK-16472 Inconsistent nullability in schema after being read
• links to: GitHub Pull Request #24176