[SPARK-7644] Ensure all scoped RDD operations are tested and cleaned

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Done
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: DStreams, Spark Core, SQL
• Target Version/s: 1.4.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 14/May/15 18:55
• Updated: 20/May/15 19:18
• Resolved: 20/May/15 19:18

Description

If all goes well, this will be a "Won't Fix". Before releasing we should make sure all operations wrapped in `RDDOperationScope.withScope` are actually tested and enclosed closures are actually cleaned. This is because a big change went into `ClosureCleaner` and wrapping methods in closures may change whether they are serializable.

TL;DR we should run all the wrapped operations to make sure we don't run into java.lang.NotSerializableException.

Issue Links

• is related to: SPARK-7741 ContextCleaner not used by many DStream operations
• is related to: SPARK-7237 Many user provided closures are not actually cleaned