[SPARK-22472] Datasets generate random values for null primitive types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1, 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: SQL
• Labels: release-notes
• Target Version/s: 2.2.1, 2.3.0
• Assignee: Wenchen Fan
• Reporter: Vladislav Kuzemchik
• Created: 08/Nov/17 15:34
• Updated: 01/Dec/17 07:22
• Resolved: 10/Nov/17 06:02

Description

Not sure if it ever were reported.

scala> val s = sc.parallelize(Seq[Option[Long]](None,Some(1L),Some(5))).toDF("v")
s: org.apache.spark.sql.DataFrame = [v: bigint]

scala> s.show(false)
+----+
|v   |
+----+
|null|
|1   |
|5   |
+----+


scala> s.as[Long].map(v => v*2).show(false)
+-----+
|value|
+-----+
|-2   |
|2    |
|10   |
+-----+


scala> s.select($"v"*2).show(false)
+-------+
|(v * 2)|
+-------+
|null   |
|2      |
|10     |
+-------+

Issue Links

• links to: [Github] Pull Request #19707 (cloud-fan)