[SPARK-5363] Spark 1.2 freeze without error notification

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1, 1.3.0
• Fix Version/s: 1.2.2, 1.3.0, 1.4.0
• Component/s: PySpark
• Target Version/s: 1.2.2, 1.3.0
• Assignee: Davies Liu
• Reporter: Tassilo Klein
• Created: 22/Jan/15 02:49
• Updated: 25/May/15 15:52
• Resolved: 26/Feb/15 19:59

Description

After a number of calls to a map().collect() statement Spark freezes without reporting any error. Within the map a large broadcast variable is used.

The freezing can be avoided by setting 'spark.python.worker.reuse = false' (Spark 1.2) or using an earlier version, however, at the prize of low speed.

Issue Links

• relates to: SPARK-3993 python worker may hang after reused from take()
• relates to: SPARK-3030 reuse python worker
• links to: [Github] Pull Request #4601 (davies)
• links to: [Github] Pull Request #4776 (davies)