[SPARK-30723] Executing the example on https://spark.apache.org/docs/latest/running-on-yarn.html fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.4
• Component/s: Spark Core, YARN
• Labels: bulk-closed

Environment

jdk1.8.0_241

hadoop-3.1.0

spark-2.4.4-bin-without-hadoop

• Assignee: Unassigned
• Reporter: Reinhard Eilmsteiner
• Created: 04/Feb/20 07:48
• Updated: 25/May/21 01:50
• Resolved: 25/May/21 01:38

Description

running

 {{./bin/spark-submit --class org.apache.spark.examples.SparkPi
--master yarn
--deploy-mode cluster
--driver-memory 4g
--executor-memory 2g
--executor-cores 1
--queue thequeue
examples/jars/spark-examples*.jar
10}}

results in

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more{{}}

Just install slf4j or is there a way around?{{}}

Issue Links

• duplicates: SPARK-30619 org.slf4j.Logger and org.apache.commons.collections classes not built as part of hadoop-provided profile