[SPARK-3902] Stabilize AsyncRDDActions and expose its methods in Java API

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Java API, Spark Core
• Target Version/s: 1.2.0
• Epic Link: Monitoring and debugging improvements (Spark 1.2)
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 10/Oct/14 21:38
• Updated: 20/Oct/14 03:02
• Resolved: 20/Oct/14 03:02

Description

The AsyncRDDActions methods are currently the easiest way to determine Spark jobs' ids for use in progress-monitoring code (see SPARK-2636). AsyncRDDActions is currently marked as @Experimental; for 1.2, I think that we should stabilize this API and expose it in Java, too.

One concern is whether there's a better async API design that we should prefer over this one as our stable API; I had some ideas for a more general API in SPARK-3626 (discussed in much greater detail on GitHub: https://github.com/apache/spark/pull/2482) but decided against the more general API due to its confusing cancellation semantics. Given this, I'd be comfortable stabilizing our current API.

Issue Links

• is related to: HIVE-7439 Spark job monitoring and error reporting [Spark Branch]
• is related to: SPARK-3590 Expose async APIs in the Java API
• is related to: SPARK-3626 Replace AsyncRDDActions with a more general async. API
• links to: [Github] Pull Request #2760 (JoshRosen)