[SPARK-15061] Upgrade Py4J to 0.10.1

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: PySpark
• External issue URL: https://github.com/bartdag/py4j/issues/201
• Assignee: Holden Karau
• Reporter: Chris Kanich
• Created: 02/May/16 13:09
• Updated: 13/May/16 08:00
• Resolved: 13/May/16 07:59

Description

Py4J 0.10.1 hasn't landed yet, but it will likely cause a significant performance improvement for PySpark and MLLib in particular. More details are available at https://github.com/bartdag/py4j/issues/201

The syscall overhead was likely the reason that https://issues.apache.org/jira/browse/SPARK-6728 was reported as well - dropping the base64 encoding will help too, but I imagine this fix will help more.

Issue Links

• links to: [Github] Pull Request #13064 (holdenk)