[SPARK-3626] Replace AsyncRDDActions with a more general async. API

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 21/Sep/14 19:12
• Updated: 10/Oct/14 21:40
• Resolved: 02/Oct/14 02:19

Description

The experimental AsyncRDDActions APIs seem to only exist in order to enable job cancellation.

We've been considering extending these APIs to support progress monitoring, but this would require stabilizing them so they're no longer @Experimental.

Instead, I propose to replace all of the AsyncRDDActions with a mechanism based on job groups which allows arbitrary computations to be run in job groups and supports cancellation / monitoring of Spark jobs launched from those computations.

(full design pending; see my GitHub PR for more details).

Issue Links

• relates to: SPARK-3902 Stabilize AsyncRDDActions and expose its methods in Java API
• links to: [Github] Pull Request #2482 (JoshRosen)