[SPARK-1700] PythonRDD leaks socket descriptors during cancellation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0, 1.0.0
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Aaron Davidson
• Reporter: Aaron Davidson
• Created: 02/May/14 22:11
• Updated: 11/Dec/14 20:32
• Resolved: 11/Dec/14 20:32

Description

Sockets from Spark to Python workers are not cleaned up over the duration of a job, causing the total number of opened file descriptors to grow to around the number of partitions in the job. Usually these go away if the job is successful, but in the case of cancellation (and possibly exceptions, though I haven't investigated), the socket file descriptors remain indefinitely.