[SPARK-18284] Scheme of DataFrame generated from RDD is different between master and 2.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.2.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Kazuaki Ishizaki
• Reporter: Kazuaki Ishizaki
• Created: 05/Nov/16 02:47
• Updated: 25/May/17 07:37
• Resolved: 02/Dec/16 04:33

Description

When the following program is executed, a schema of dataframe is different among master, branch 2.0, and branch 2.1. The result should be false.

val df = sparkContext.parallelize(1 to 8, 1).toDF()
df.printSchema
df.filter("value > 4").count

=== master ===
root
 |-- value: integer (nullable = true)

=== branch 2.1 ===
root
 |-- value: integer (nullable = true)

=== branch 2.0 ===
root
 |-- value: integer (nullable = false)

Issue Links

• is duplicated by: SPARK-20866 Dataset map does not respect nullable field
• is related to: SPARK-18623 Add `returnNullable` to `StaticInvoke` and modify it to handle properly.
• supercedes: SPARK-14584 Improve recognition of non-nullability in Dataset transformations
• links to: [Github] Pull Request #15780 (kiszk)