[SPARK-41188] Set executorEnv OMP_NUM_THREADS to be spark.task.cpus by default for spark executor JVM processes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.2.3, 3.3.2, 3.4.0
• Component/s: ML, Spark Core
• Assignee: Weichen Xu
• Reporter: Weichen Xu
• Created: 18/Nov/22 03:16
• Updated: 28/Feb/23 02:05
• Resolved: 19/Nov/22 09:27

Description

Set executorEnv OMP_NUM_THREADS to be spark.task.cpus by default for spark executor JVM processes, this is for limiting the thread number for OpenBLAS routine to the number of cores assigned to this executor because some spark ML algorithms calls OpenBlAS via netlib-java

Issue Links

• causes: SPARK-42596 [YARN] OMP_NUM_THREADS not set to number of executor cores by default
• relates to: SPARK-42613 PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default
• links to: [Github] Pull Request #38699 (WeichenXu123)