[SPARK-24405] parameter for python worker timeout

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: PySpark
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Deepansh
• Created: 28/May/18 21:38
• Updated: 08/Oct/19 05:43
• Resolved: 08/Oct/19 05:43

Description

Currently, the python worker timeout is set to 60000 ms in code and cannot be changed. I am developing a pure python application, but workers die after 1min of inactivity. And new activity after 1min makes new python workers and sets the environment again. Can we make this parameter configurable with the default being set to 1min so that pure python applications are faster?

parameter = "IDLE_WORKER_TIMEOUT_MS"
file =PythonWorkerFactory.scala