[SPARK-2737] ClassCastExceptions when collect()ing JavaRDDs' underlying Scala RDDs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.9.0, 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Java API
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 29/Jul/14 21:14
• Updated: 23/Sep/15 08:21
• Resolved: 31/Jul/14 05:42

Description

The Java API's use of fake ClassTags doesn't seem to cause any problems for Java users, but it can lead to issues when passing JavaRDDs' underlying RDDs to Scala code (e.g. in the MLlib Java API wrapper code). If we call collect() on a Scala RDD with an incorrect ClassTag, this causes ClassCastExceptions when we try to allocate an array of the wrong type (for example, see SPARK-2197).

There are a few possible fixes here. An API-breaking fix would be to completely remove the fake ClassTags and require Java API users to pass java.lang.Class instances to all parallelize() calls and add returnClass fields to all Function implementations. This would be extremely verbose.

Instead, I propose that we add internal APIs to "repair" a Scala RDD with an incorrect ClassTag by wrapping it and overriding its ClassTag. This should be okay for cases where the Scala code that calls collect() knows what type of array should be allocated, which is the case in the MLlib wrappers.

Issue Links

• is related to: SPARK-4489 JavaPairRDD.collectAsMap from checkpoint RDD may fail with ClassCastException
• is required by: SPARK-2197 Spark invoke DecisionTree by Java
• links to: [Github] Pull Request #1639 (JoshRosen)