[SPARK-43782] Support log level configuration with static Spark conf

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Yan-Lin (Jared) Wang
• Reporter: Yan-Lin (Jared) Wang
• Created: 24/May/23 23:56
• Updated: 31/May/23 06:08
• Resolved: 31/May/23 06:06

Description

Today users are using sc.setLogLevel to configure the current log level using SparkContext accessible from within the SparkSession. It is a convenient alternative to directly depending on the Log4j mechanism like statically changing the log4j.xml or dynamically invoke the Log4j API.

It would be great to allow the configuration of the log level using a static Spark conf that has the same kind of simplicity, which comes in handy when we don't want to manually change the log4j.xml. This is particularly helpful in the Spark Connect scenario where there’s no way for the client to call setLogLevel because SparkContext is not yet supported in its API, and when it connects to a platform where it can't easily change the Log4j config file.
./bin/pyspark --conf spark.log.level ERROR

Issue Links

• links to: [Github] Pull Request #41302 (jwang0306)
• links to: [Github] Pull Request #41302 (jwang0306)