[SPARK-2758] UnionRDD's UnionPartition should not reference parent RDDs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.0.1, 1.0.2
• Fix Version/s: 1.1.0
• Target Version/s: 1.1.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 30/Jul/14 23:20
• Updated: 31/Jul/14 04:30
• Resolved: 31/Jul/14 04:30

Description

UnionPartition has a non-transient field referencing the parent RDD, to be used in compute (iterator). That causes some trouble with task size because partition objects are supposed to be small.

Issue Links

• blocks: SPARK-2585 Remove special handling of Hadoop JobConf
• links to: [Github] Pull Request #1675 (rxin)