Avro
  1. Avro
  2. AVRO-1103

New AvroDeserializer should Locate Appropriate Classloader

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.7.2
    • Component/s: java
    • Labels:
      None
    • Environment:

      Hadoop 0.23.1 with Avro jars replaced by 1.7 jars
      Specific data classes assembled into JAR with mapper/reducer

    • Tags:
      classloader

      Description

      Continuing on from AVRO-873 I believe some more work needs to be done to get the MapReduce 2 APIs in Avro 1.7 working with Hadoop 0.23. Since it revolves around classloaders it is complex to present a unit test which fails so I will explain the problem:

      • By default SpecificDatumReader will use the classloader it was loaded from to find a Specific class to deserialize into.
      • In earlier versions of Hadoop e.g. 0.20.2 Avro was not included so typically you would bundle Avro into your job jar along with the Specific classes so they would be on the same classpath.
      • However later versions of Hadoop such as 0.23 ship with Avro. Thus you find that the SpecificData.class.getClassloader() is typically a parent loader which just contains Hadoop components.
      • Thus when SpecificData goes to construct a Specific class from the schema it cannot locate it and silently defaults to creating a GenericData.

      In AVRO-873 an additional constructor was added to SpecificData to force it to use a different classloader. Thus to extend this fix to the new MR2 APIs:

      • AvroDeserializer could attempt to instantiate the class using Class.forName() and from this get the appropriate Classloader and pass this into the constructor of SpecificDatumReader.
      • Line 2771 of SpecificData.java is:

      Class c = SpecificData.get().getClass(schema);

      • This would need to be changed to:

      Class c = this.getClass(schema);

      I have raised this in the mail groups here: http://search-hadoop.com/m/wVUf1aLCwd/classloader/v=threaded so apologies if this is already being thought about.

      1. AVRO-1103.patch
        11 kB
        Doug Cutting
      2. AVRO-1103.patch
        8 kB
        Doug Cutting
      3. AVRO-1103.patch
        7 kB
        Doug Cutting
      4. AVRO-1103-for 0.23.1.patch
        22 kB
        Jacob Metcalf
      5. AvroCDH4.patch
        33 kB
        Jacob Metcalf

        Issue Links

          Activity

            People

            • Assignee:
              Doug Cutting
              Reporter:
              Jacob Metcalf
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development