[SPARK-27041] large partition data cause pyspark with python2.x oom

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: PySpark
• Assignee: David Yang
• Reporter: David Yang
• Created: 04/Mar/19 09:02
• Updated: 12/Mar/19 15:25
• Resolved: 12/Mar/19 15:24

Description

With large partition, pyspark may exceeds executor memory limit and trigger out of memory for python 2.7.
This is because map() is used. Unlike in python3.x, python 2.7 map() will generate a list and need to read all data into memory.

The proposed fix will use imap in python 2.7 and it has been verified.

Issue Links

• links to: GitHub Pull Request #23954