[SPARK-5091] Hooks for PySpark tasks

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: PySpark
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Davies Liu
• Created: 05/Jan/15 20:21
• Updated: 21/May/19 05:37
• Resolved: 21/May/19 05:37

Description

Currently, it's not convenient to add package on executor to PYTHONPATH (we did not assume the environment of driver an executor are identical).

It will be nice to have a hook to called before/after every tasks, then user could manipulate sys.path by pre-task-hooks.