Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.19.0
-
None
-
None
-
Reviewed
Description
I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
Caused by: java.io.IOException: WritableName can't load class at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73) at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596) ... 33 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71) ... 34 more
I'll attach a unit test that can demonstrate this more clearly....
Attachments
Attachments
Issue Links
- relates to
-
HADOOP-6103 Configuration clone constructor does not clone all the members.
- Closed