[SPARK-927] PySpark sample() doesn't work if numpy is installed on master but not on workers

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.9.1, 1.0.2, 1.1.2
• Fix Version/s: 1.2.0
• Component/s: PySpark
• Assignee: Matthew Farrellee
• Reporter: Josh Rosen
• Created: 08/Oct/13 11:03
• Updated: 05/Jan/15 23:05
• Resolved: 05/Jan/15 23:05

Description

PySpark's sample() method crashes with ImportErrors on the workers if numpy is installed on the driver machine but not on the workers. I'm not sure what's the best way to fix this. A general mechanism for automatically shipping libraries from the master to the workers would address this, but that could be complicated to implement.

Issue Links

• is related to: SPARK-4477 remove numpy from RDDSampler of PySpark
• links to: [Github] Pull Request #2313 (mattf)