[SPARK-10714] Refactor PythonRDD to decouple iterator computation from PythonRDD

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: PySpark, Spark Core
• Target Version/s: 1.6.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 19/Sep/15 07:28
• Updated: 27/Jan/16 04:20
• Resolved: 22/Sep/15 21:22

Description

The idea is that most of the logic of calling Python actually has nothing to do with RDD (it is really just communicating with a socket – there is nothing distributed about it), and it is only currently depending on RDD because it was written this way.

If we extract that functionality out, we can apply it to area of the code that doesn't depend on RDDs, and also make it easier to test.

Issue Links

• is duplicated by: SPARK-10494 Multiple Python UDFs together with aggregation or sort merge join may cause OOM (failed to acquire memory)
• relates to: SPARK-12792 Refactor RRDD to support R UDF
• links to: [Github] Pull Request #8835 (rxin)