[SPARK-23971] Should not leak Spark sessions across test suites

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: SQL, Tests
• Assignee: Eric Liang
• Reporter: Eric Liang
• Created: 12/Apr/18 19:46
• Updated: 02/May/18 19:02
• Resolved: 13/Apr/18 05:31

Description

Many suites currently leak Spark sessions (sometimes with stopped SparkContexts) via the thread-local active Spark session and default Spark session. We should attempt to clean these up and detect when this happens to improve the reproducibility of tests.

Issue Links

• links to: [Github] Pull Request #21058 (ericl)
• links to: [Github] Pull Request #21197 (gatorsmile)