[SPARK-5223] Use pickle instead of MapConvert and ListConvert in MLlib Python API

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.1, 1.3.0
• Component/s: MLlib, PySpark
• Target Version/s: 1.2.1, 1.3.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 13/Jan/15 18:21
• Updated: 13/Jan/15 20:51
• Resolved: 13/Jan/15 20:50

Description

It will introduce problems if the object in dict/list/tuple can not support by py4j, such as Vector.

Also, pickle may have better performance for larger object (less RPC).

In some cases that the object in dict/list can not be pickled (such as JavaObject), we should still use MapConvert/ListConvert.

discussion: http://apache-spark-developers-list.1001551.n3.nabble.com/Python-to-Java-object-conversion-of-numpy-array-td10065.html

Issue Links

• links to: [Github] Pull Request #4023 (davies)