[SPARK-3993] python worker may hang after reused from take()

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: PySpark
• Target Version/s: 1.2.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 17/Oct/14 21:05
• Updated: 06/Feb/15 10:23
• Resolved: 24/Oct/14 00:21

Description

After take(), maybe there are some garbage left in the socket, then next task assigned to this worker will hang because of corrupted data.

We should make sure the socket is clean before reuse it.

Issue Links

• is related to: SPARK-5363 Spark 1.2 freeze without error notification
• links to: [Github] Pull Request #2838 (davies)