[SPARK-15964] Assignment to RDD-typed val fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.0.0

Environment

Notebook on Databricks Community-Edition
Spark-2.0 preview
Google Chrome Browser
Linux Ubuntu 14.04 LTS

• Assignee: Unassigned
• Reporter: Sanjay Dasgupta
• Created: 15/Jun/16 14:48
• Updated: 15/Jun/16 18:08
• Resolved: 15/Jun/16 18:08

Description

Unusual assignment error, giving the following error message:

found : org.apache.spark.rdd.RDD[Name]
required : org.apache.spark.rdd.RDD[Name]

This occurs when the assignment is attempted in a cell that is different from the cell in which the item on the right-hand-side is defined. As in the following example:

// CELL-1
import org.apache.spark.sql.Dataset
import org.apache.spark.rdd.RDD

case class Name(number: Int, name: String)
val names = Seq(Name(1, "one"), Name(2, "two"), Name(3, "three"), Name(4, "four"))
val dataset: Dataset[Name] = spark.sparkContext.parallelize(names).toDF.as[Name]

// CELL-2
// Error reported here ...
val dataRdd: RDD[Name] = dataset.rdd

The error is reported in CELL-2

Issue Links

• relates to: SPARK-1199 Type mismatch in Spark shell when using case class defined in shell