[SPARK-24585] Adding ability to audit file system before and after test to ensure all files are cleaned up.

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: Build
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: David Lewis
• Created: 18/Jun/18 22:07
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

Some spark tests use temporary files and folders on the file system. This proposal is to audit the file system before and after to ensure that all files created are cleaned up.

This proposal is to add two flags to SparkFunSuite. One to enable the auditing, which will log a warning if there is a discrepancy, and one to throw an exception if there is a discrepancy. 

This is intended to help prevent file leakage in spark.