Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
We made some improvements in SPARK-3543 but for Spark 1.2 we should convert TaskContext into a fully stable API. To do this I’d suggest the following changes - note that some of this reverses parts of SPARK-3543. The goal is to provide a class that users can’t easily construct and exposes only the public functionality.
1. Separate TaskContext into a public abstract class (TaskContext) and a private implementation called TaskContextImpl. The former should be a Java abstract class - the latter should be a private[spark] Scala class to reduce visibility (or maybe we can keep it as Java and tell people not to use it?).
2. TaskContext abstract class will have (NOTE: this changes getXX() to XX() intentionally)
public isCompleted()
public isInterrupted()
public addTaskCompletionListener(...)
public addTaskCompletionCallback(...) (deprecated)
public stageId()
public partitionId()
public attemptId()
pubic isRunningLocally()
STATIC
public get()
set() and unset() at default visibility
3. A new private[spark] static object TaskContextHelper in the same package as TaskContext will exist to expose set() and unset() from within Spark using forwarder methods that just call TaskContext.set(). If someone within Spark wants to set this they call TaskContextHelper.set() and it forwards it.
4. TaskContextImpl will be used whenever we construct a TaskContext internally.
