[SPARK-17756] java.lang.ClassCastException when using cartesian with DStream.transform

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.3.0
• Fix Version/s: 2.4.0
• Component/s: DStreams, PySpark
• External issue URL: http://stackoverflow.com/q/39804337/1560062
• Assignee: Hyukjin Kwon
• Reporter: Maciej Szymkiewicz
• Created: 01/Oct/16 19:37
• Updated: 12/Dec/22 18:10
• Resolved: 08/Jun/18 18:28

Description

Steps to reproduce:

from pyspark.streaming import StreamingContext

ssc = StreamingContext(spark.sparkContext, 10)
(ssc
    .queueStream([sc.range(10)])
    .transform(lambda rdd: rdd.cartesian(rdd))
    .pprint())

ssc.start()

## 16/10/01 21:34:30 ERROR JobScheduler: Error generating jobs for time 1475350470000 ms
## java.lang.ClassCastException: org.apache.spark.api.java.JavaPairRDD ## cannot be cast to org.apache.spark.api.java.JavaRDD
## 	at com.sun.proxy.$Proxy15.call(Unknown Source)
##    ....

A dummy fix is to put map(lamba x: x) which suggests it is a problem similar to https://issues.apache.org/jira/browse/SPARK-16589

Issue Links

• relates to: SPARK-16589 Chained cartesian produces incorrect number of records
• links to: [Github] Pull Request #19498 (HyukjinKwon)