[SPARK-50073] Intended way of using log4j plugin JARs

• Type: Question
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.5.3
• Component/s: Documentation
• Assignee: Unassigned
• Reporter: Nikita Sokolov
• Created: 22/Oct/24 13:53
• Updated: 22/Oct/24 13:55

Description

I want to use a log4j plugin, which would append the logs of Master JVM using classes from some 3rd party JARs:
https://logging.apache.org/log4j/2.x/manual/plugins.html

The Spark configuration documentation says that I should configure the log4j via conf/log4j2.properties.
https://spark.apache.org/docs/latest/configuration.html#configuring-logging

This would require me to add my JARs on the class-path of Master, but it is not customizable and always is $SPARK_HOME/jars:

https://github.com/apache/spark/blob/master/bin/spark-class#L71

Is it intended to add such JARs to the $SPARK_HOME/jars?