[SPARK-38502] Distribution with hadoop-provided is missing log4j2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.3.0
• Component/s: Build
• Target Version/s: 3.3.0
• Assignee: Unassigned
• Reporter: Emil Ejbyfeldt
• Created: 10/Mar/22 12:40
• Updated: 11/Mar/22 07:03
• Resolved: 11/Mar/22 07:03

Description

Currently building spark 3.3.0-SNAPSHOT using `./dev/make-distribution.sh --tgz --name hadoop-provided-test -Phadoop-provided -Pyarn` script will build a package that does not included log4j2. Trying to run spark-submit with the latest hadoop release 3.3.2 and this build will result in

$ spark-submit run-example org.apache.spark.examples.SparkPi
Error: Unable to initialize main class org.apache.spark.deploy.SparkSubmit
Caused by: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Filter

Since log4j2 is not found. So I believe the maven build settings needs to be tweaked so that log4j2 is included in the spark distribution.