[SPARK-5224] parallelize list/ndarray is really slow

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.1, 1.3.0
• Component/s: PySpark
• Target Version/s: 1.2.1, 1.3.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 13/Jan/15 18:53
• Updated: 15/Jan/15 19:41
• Resolved: 15/Jan/15 19:41

Description

After the default batchSize changed to 0 (batched based on the size of object), but parallelize() still use BatchedSerializer with batchSize=1.

Also, BatchedSerializer did not work well with list and numpy.ndarray

Issue Links

• links to: [Github] Pull Request #4024 (davies)