[SPARK-6055] Memory leak in pyspark sql due to incorrect equality check

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.1, 1.2.1, 1.3.0
• Fix Version/s: 1.1.2, 1.2.2, 1.3.0
• Component/s: PySpark, SQL
• Target Version/s: 1.1.2, 1.2.2, 1.3.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 27/Feb/15 07:28
• Updated: 28/Feb/15 04:09
• Resolved: 28/Feb/15 04:09

Description

The _eq_ of DataType is not correct, class cache is not used correctly (created class can not be find by dataType), then it will create lots of classes (saved in _cached_cls), never released.

Also, all same DataType have same hash code, there will be many object in a dict with the same hash code, end with hash attach, it's very slow to access this dict (depends on the implementation of CPython).

Issue Links

• links to: [Github] Pull Request #4808 (davies)
• links to: [Github] Pull Request #4809 (davies)
• links to: [Github] Pull Request #4810 (davies)