[SPARK-3543] Write TaskContext in Java and expose it through a static accessor

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Prashant Sharma
• Reporter: Patrick Wendell
• Created: 16/Sep/14 04:49
• Updated: 03/Feb/15 06:17
• Resolved: 27/Sep/14 04:30

Description

Right now we have these xWithContext methods and it's a bit awkward (for instance, we don't support accessing taskContext from a normal map or filter operation). I'd propose the following

1. Re-write TaskContext in Java - it's a simple class. It can still refer to the scala version of TaskMetrics.
2. Have a static method `TaskContext.get()` which will return the current in-scope TaskContext. Under the hood this uses a thread local variable similar to SparkEnv that the Executor sets.
3. Deprecate all of the existing xWithContext methods.

Issue Links

• contains: SPARK-2895 Support mapPartitionsWithContext in Spark Java API
• is related to: SPARK-5549 Define TaskContext interface in Scala
• links to: [Github] Pull Request #2425 (ScrapCodes)
• links to: [Github] Pull Request #2557 (rxin)
• links to: [Github] Pull Request #2560 (rxin)