[SPARK-3679] pickle the exact globals of functions

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: PySpark
• Target Version/s: 1.2.0
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 24/Sep/14 18:28
• Updated: 24/Sep/14 20:00
• Resolved: 24/Sep/14 20:00

Description

function.func_code.co_names has all the names used in the function, including name of attributes. It will pickle some unnecessary globals if there is a global having the same name with attribute (in co_names).

There is a regression introduced by PR 2114 https://github.com/apache/spark/pull/2144/files

Issue Links

• links to: [Github] Pull Request #2522 (davies)