[SPARK-44198] Support propagation of the log level to the executors

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Assignee: Vinod KC
• Reporter: Vinod KC
• Created: 26/Jun/23 17:47
• Updated: 27/Jul/23 23:44
• Resolved: 27/Jul/23 23:44

Description

Currently, the sc.setLogLevel() method only sets the log level on the Spark driver, failing to reflect the desired log level on the executors. This inconsistency can lead to difficulties in debugging and monitoring Spark applications, as log messages from the executors may not align with the expected log level set on the user code.

Issue Links

• links to: [Github] Pull Request #41746 (vinodkc)
• links to: [Github] Pull Request #41746 (vinodkc)