Minor Description
Below is the reproduced code.
import org.apache.spark.sql.Encoders case class TestSchema(x: Int, y: Int) case class BaseSchema(value: TestSchema) val schema = Encoders.product[BaseSchema].schema val testDS = Seq("""{"value":{"x":1}}""", """{"value":{"x":2}}""").toDS val jsonDS = spark.read.schema(schema).json(testDS) jsonDS.show +---------+ | value| +---------+ |{1, null}| |{2, null}| +---------+ jsonDS.cache.show +------+ | value| +------+ |{1, 0}| |{2, 0}| +------+
The above result occurs when a schema is created with a nested StructType and nullable of StructField is false.
Attachments
Issue Links
- links to
(2 links to)