Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4975

CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.

      However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:

      Caused by: java.io.IOException: WritableName can't load class
      	at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
      	at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
      	... 33 more
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
      	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:247)
      	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
      	at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
      	... 34 more
      

      I'll attach a unit test that can demonstrate this more clearly....

        Attachments

        1. break-wrapped-rr-test.patch
          6 kB
          Jingkei Ly
        2. HADOOP-4975-1.patch
          0.8 kB
          Jingkei Ly
        3. HADOOP-4975-2.patch
          7 kB
          Jingkei Ly

          Issue Links

            Activity

              People

              • Assignee:
                jly Jingkei Ly
                Reporter:
                jly Jingkei Ly
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: