[HDDS-781] Ambari HDP NoClassDefFoundError for MR jobs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.3.0
• Component/s: Ozone Filesystem
• Assignee: Unassigned
• Reporter: Soumitra Sulav
• Created: 31/Oct/18 21:18
• Updated: 01/Jun/20 01:32
• Resolved: 01/Jun/20 01:32

Description

HDP integrated with Ambari has a /usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz file containing all the libraries needed for a MR job to run and is copied in the yarn containers at time of execution.

As introducing ozone filesystem, relevant jars need to be packaged as part of the tar, also the tar is placed as part of yum install hadoop components done by Ambari during cluster setup.

During an MR Job run, I faced below java.lang.NoClassDefFoundError exceptions :

org/apache/hadoop/fs/ozone/OzoneFileSystem
org/apache/ratis/proto/RaftProtos$ReplicationLevel
org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum

 

Adding the relevant jar in the mentioned tar file resolves the exception.

Complete stacktrace for one of the NoClassDefFoundError exception :

2018-10-31 10:03:05,191 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoClassDefFoundError: org/apache/ratis/proto/RaftProtos$ReplicationLevel
 at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64)
 at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221)
 at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116)
 at org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421)
 at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214)
 at org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127)
 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
 at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
 at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160)
 at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116)
 at org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134)
 at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35)
 at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.ClassNotFoundException: org.apache.ratis.proto.RaftProtos$ReplicationLevel
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 ... 29 more
2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.NoClassDefFoundError: org/apache/ratis/proto/RaftProtos$ReplicationLevel