[SPARK-3458] enable use of python's "with" statements for SparkContext management

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: PySpark
• Labels: features python sparkcontext
• Assignee: Matthew Farrellee
• Reporter: Matthew Farrellee
• Created: 09/Sep/14 15:51
• Updated: 02/Mar/21 06:09
• Resolved: 10/Sep/14 01:55

Description

best practice for managing SparkContexts involves exception handling, e.g.

try:
  sc = SparkContext()
  app(sc)
finally:
  sc.stop()

python provides the "with" statement to simplify this code, e.g.

with SparkContext() as sc:
  app(sc)

the SparkContext should be usable in a "with" statement

Issue Links

• is related to: SPARK-2972 APPLICATION_COMPLETE not created in Python unless context explicitly stopped
• is related to: SPARK-3470 Have JavaSparkContext implement Closeable/AutoCloseable
• is related to: SPARK-3471 Automatic resource manager for SparkContext in Scala?
• links to: [Github] Pull Request #2335 (mattf)
• links to: [Github] Pull Request #31698 (LuciferYang)
• links to: [Github] Pull Request #31698 (LuciferYang)