[SPARK-42613] PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.5.0
• Component/s: PySpark, YARN
• Assignee: John Zhuge
• Reporter: John Zhuge
• Created: 28/Feb/23 02:03
• Updated: 02/Mar/23 00:18
• Resolved: 02/Mar/23 00:18

Description

Follow up from https://github.com/apache/spark/pull/40199#discussion_r1119453996

If OMP_NUM_THREADS is not set explicitly, we should set it to `spark.task.cpus` instead of `spark.executor.cores` as described in PR #38699.

Issue Links

• is related to: SPARK-41188 Set executorEnv OMP_NUM_THREADS to be spark.task.cpus by default for spark executor JVM processes
• is related to: SPARK-42596 [YARN] OMP_NUM_THREADS not set to number of executor cores by default
• is related to: SPARK-28843 Set OMP_NUM_THREADS to executor cores reduce Python memory consumption
• links to: [Github] Pull Request #40212 (jzhuge)