[OOZIE-3598] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.1.0
• Component/s: scripts
• Assignee: Unassigned
• Reporter: TomQeeMoer
• Created: 22/Apr/20 09:09
• Updated: 22/Apr/20 09:09

Description

This is my first time to try oozie,I followed the steps in quick start:

prepare step: copy the libs into libext

 

mkdir libext
cp /opt/hadoop/share/hadoop/*/*.jar libext/
cp /opt/hadoop/share/hadoop/*/lib/*.jar libext/
cp /tmp/ext-2.2.zip libext/

 

step1:  upload the sharelib

./bin/oozie-setup.sh sharelib create -fs hdfs://hadoop01/ -locallib oozie-sharelib-5.1.0.tar.gz

step2: create database

 

 

./bin/ooziedb.sh create -sqlfile oozie.sql -run

step3:  run oozied

 

 

./bin/oozied.sh run

Error occured:

 

 

Error: A JNI error has occurred, please check your installation and try againError: A JNI error has occurred, please check your installation and try againException in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ... 7 more

 

 

I checked the jar files for many times, and I confirmed that the class org.apache.hadoop.conf.Configuration was included within hadoop-common-2.7.7.jar

now I find out why:

when I run step1, the oozie-setup.sh scipt , code in lines 136~140,create the 'lib' directory, and copied the self contained jars  in ${JETTY_LIB_DIR} to lib

 

#Create lib directory from war if lib doesn't exist    
if [ ! -d "${BASEDIR}/lib" ]; then
      mkdir ${BASEDIR}/lib      
      cp ${JETTY_LIB_DIR}/*  ${BASEDIR}/lib
fi

 

 

when I run step3, the function prepare_jetty()  in ** oozie-setup.sh scipt copied the jars in libext to  ${JETTY_LIB_DIR},there is some mistakes in lines 236-237:

 

found=`ls ${JETTY_LIB_DIR}/${jarPath} 2> /dev/null | wc -l`
checkExec "looking for JAR ${jarPath} in ${JETTY_LIB_DIR}"

1. when the ${JETTY_LIB_DIR} is /opt/oozie-5.1.0/embedded-oozie-server/webapp/WEB-INF/lib and the ${jarPath} is /opt/oozie-5.1.0/libext/somejar.jar,

 

the Combination ${JETTY_LIB_DIR}/${jarPath} is mistaken

I thought it should be like this:

found=`ls ${JETTY_LIB_DIR}/${jarPath##*/} 2> /dev/null | wc -l`

2. Since the return status of statement "ls ${JETTY_LIB_DIR}/${jarPath} 2> /dev/null | wc -l" is always 0, the error message "looking for JAR ${jarPath} in ${JETTY_LIB_DIR}" will never be printed.

 

In any case, the jars in libext were copied into ${JETTY_LIB_DIR} successfully. But ${JETTY_LIB_DIR}  in NOT included in "-Djava.library.path -cp"

code in oozie-jetty-server.sh,lines 81-83

 

jetty_opts="${jetty_opts} -Djava.library.path=${JAVA_LIBRARY_PATH}";
jetty_opts="${jetty_opts} -cp ${JETTY_DIR}/*:${JETTY_DIR}/dependency/*:${BASEDIR}/lib/*:${BASEDIR}/libtools/*:${JETTY_DIR}"

 

and the function symlink_lib() do NOT work , since the 'lib' directory was created in step1,which is not including the hadoop jars.

symlink_lib() {  test -e ${BASEDIR}/lib || ln -s ${JETTY_DIR}/webapp/WEB-INF/lib ${BASEDIR}/lib}

 

I modified the oozie-setup.sh, ** and it worked for me . see it attachement

Forgive my poor english