Avro
  1. Avro
  2. AVRO-1103

New AvroDeserializer should Locate Appropriate Classloader

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.7.2
    • Component/s: java
    • Labels:
      None
    • Environment:

      Hadoop 0.23.1 with Avro jars replaced by 1.7 jars
      Specific data classes assembled into JAR with mapper/reducer

    • Tags:
      classloader

      Description

      Continuing on from AVRO-873 I believe some more work needs to be done to get the MapReduce 2 APIs in Avro 1.7 working with Hadoop 0.23. Since it revolves around classloaders it is complex to present a unit test which fails so I will explain the problem:

      • By default SpecificDatumReader will use the classloader it was loaded from to find a Specific class to deserialize into.
      • In earlier versions of Hadoop e.g. 0.20.2 Avro was not included so typically you would bundle Avro into your job jar along with the Specific classes so they would be on the same classpath.
      • However later versions of Hadoop such as 0.23 ship with Avro. Thus you find that the SpecificData.class.getClassloader() is typically a parent loader which just contains Hadoop components.
      • Thus when SpecificData goes to construct a Specific class from the schema it cannot locate it and silently defaults to creating a GenericData.

      In AVRO-873 an additional constructor was added to SpecificData to force it to use a different classloader. Thus to extend this fix to the new MR2 APIs:

      • AvroDeserializer could attempt to instantiate the class using Class.forName() and from this get the appropriate Classloader and pass this into the constructor of SpecificDatumReader.
      • Line 2771 of SpecificData.java is:

      Class c = SpecificData.get().getClass(schema);

      • This would need to be changed to:

      Class c = this.getClass(schema);

      I have raised this in the mail groups here: http://search-hadoop.com/m/wVUf1aLCwd/classloader/v=threaded so apologies if this is already being thought about.

      1. AVRO-1103.patch
        7 kB
        Doug Cutting
      2. AVRO-1103.patch
        8 kB
        Doug Cutting
      3. AVRO-1103.patch
        11 kB
        Doug Cutting
      4. AVRO-1103-for 0.23.1.patch
        22 kB
        Jacob Metcalf
      5. AvroCDH4.patch
        33 kB
        Jacob Metcalf

        Issue Links

          Activity

          Brock Noland made changes -
          Link This issue is related to AVRO-1240 [ AVRO-1240 ]
          Doug Cutting made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.7.2 [ 12322476 ]
          Resolution Fixed [ 1 ]
          Hide
          Doug Cutting added a comment -

          I committed this.

          Show
          Doug Cutting added a comment - I committed this.
          Hide
          Doug Cutting added a comment -

          I'll commit this soon unless someone objects.

          Show
          Doug Cutting added a comment - I'll commit this soon unless someone objects.
          Hide
          Steven Willis added a comment -

          I just came from AVRO-1123 which addresses a subset of this issue. I'm still having problems with avro when my generated class can only be found via an alternate ClassLoader. This patch should fix my issues. I'd appreciate it if this jira isn't closed, but the patch applied.

          A simple way to test the issue I'm having is to put a generated avro class in a jar along with another class which tries to read from an avro file containing records of your class, then run:

          $ hadoop jar myjar.jar MainClass input_file.avro

          You'll get a generic back from the reader instead of your generated class because RunJar uses an alternate ClassLoader to run the jar. However if you run:

          $ HADOOP_CLASSPATH=myjar.jar hadoop jar myjar.jar MainClass input_file.avro

          Then you'll get what you expect because now the jar containing the avro class is also accessible via the standard ClassLoader since it's now on the classpath.

          Show
          Steven Willis added a comment - I just came from AVRO-1123 which addresses a subset of this issue. I'm still having problems with avro when my generated class can only be found via an alternate ClassLoader. This patch should fix my issues. I'd appreciate it if this jira isn't closed, but the patch applied. A simple way to test the issue I'm having is to put a generated avro class in a jar along with another class which tries to read from an avro file containing records of your class, then run: $ hadoop jar myjar.jar MainClass input_file.avro You'll get a generic back from the reader instead of your generated class because RunJar uses an alternate ClassLoader to run the jar. However if you run: $ HADOOP_CLASSPATH=myjar.jar hadoop jar myjar.jar MainClass input_file.avro Then you'll get what you expect because now the jar containing the avro class is also accessible via the standard ClassLoader since it's now on the classpath.
          Doug Cutting made changes -
          Link This issue is related to AVRO-1123 [ AVRO-1123 ]
          Doug Cutting made changes -
          Fix Version/s 1.7.1 [ 12321552 ]
          Hide
          Jacob Metcalf added a comment -

          If you are okay with it I am going to close this issue. I have had to switch my environment to Hadoop CDH4 under which I cannot reproduce the problem (reasons below). I have also attached my latest patch for building Avro 1.7 with CDH4 in case its of any use when you come to supporting Hadoop 2.

          In terms of why I cannot reproduce it on CDH4. I was getting this problem because the core Avro jar and the jar with my Avro specific classes in were being loaded by different class loaders. This was because I was distributing my Avro Specific classes via the /lib mechanism - option 2 on http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/. However as this page alludes to Cloudera are phasing this option out so now I have now switched to option 3 - just dropping all my jars directly into the lib directory on all nodes. Hence they all get loaded by the same classloader and everything works.

          Show
          Jacob Metcalf added a comment - If you are okay with it I am going to close this issue. I have had to switch my environment to Hadoop CDH4 under which I cannot reproduce the problem (reasons below). I have also attached my latest patch for building Avro 1.7 with CDH4 in case its of any use when you come to supporting Hadoop 2. In terms of why I cannot reproduce it on CDH4. I was getting this problem because the core Avro jar and the jar with my Avro specific classes in were being loaded by different class loaders. This was because I was distributing my Avro Specific classes via the /lib mechanism - option 2 on http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/ . However as this page alludes to Cloudera are phasing this option out so now I have now switched to option 3 - just dropping all my jars directly into the lib directory on all nodes. Hence they all get loaded by the same classloader and everything works.
          Jacob Metcalf made changes -
          Attachment AvroCDH4.patch [ 12534276 ]
          Hide
          Jacob Metcalf added a comment -

          I have not yet delved far enough into the innards of Hadoop to understand why Configuration.getClassloader() does not work for me. I can see it does the following:

          private ClassLoader classLoader;
          {{
          classLoader = Thread.currentThread().getContextClassLoader();
          if (classLoader == null) {{
          classLoader = Configuration.class.getClassLoader();
          }}
          }}

          When I debug 0.23.1 in standalone mode I can see that the config classLoader does not reference the jar containing my Avro specific classes from the /lib directory of my job jar. I agree calling Class.forName(...).getClassLoader() is a sledge hammer to crack a nut. I can debug a bit more this week to try and work out why.

          For Hadoop 2 its both pom and code changes. But the code changes were relatively easy as the difference in the versions seemed to be confined to TaskAttemptContext & SequenceFileBase.

          Show
          Jacob Metcalf added a comment - I have not yet delved far enough into the innards of Hadoop to understand why Configuration.getClassloader() does not work for me. I can see it does the following: private ClassLoader classLoader; {{ classLoader = Thread.currentThread().getContextClassLoader(); if (classLoader == null) {{ classLoader = Configuration.class.getClassLoader(); }} }} When I debug 0.23.1 in standalone mode I can see that the config classLoader does not reference the jar containing my Avro specific classes from the /lib directory of my job jar. I agree calling Class.forName(...).getClassLoader() is a sledge hammer to crack a nut. I can debug a bit more this week to try and work out why. For Hadoop 2 its both pom and code changes. But the code changes were relatively easy as the difference in the versions seemed to be confined to TaskAttemptContext & SequenceFileBase.
          Hide
          Doug Cutting added a comment -

          (I'm mostly offline this week, so will be slow to respond.)

          > your patched AvroSerialization whilst it uses the config object is still
          > trying to use the parent ClassLoader so cannot find my Avro class

          I'm confused by this. Using the classloader from the configuration is what Hadoop does for Writables, no? If the configuration has the wrong classloader then we might set that to be the same as what Class.forName() uses. Calling Class.forName.getClassloader() seems circular to me.

          > it would help me immensely if a version of avro-mapred 1.7 for Hadoop 2 could be made available

          Are code changes required or simply changes in pom.xml dependencies?

          Show
          Doug Cutting added a comment - (I'm mostly offline this week, so will be slow to respond.) > your patched AvroSerialization whilst it uses the config object is still > trying to use the parent ClassLoader so cannot find my Avro class I'm confused by this. Using the classloader from the configuration is what Hadoop does for Writables, no? If the configuration has the wrong classloader then we might set that to be the same as what Class.forName() uses. Calling Class.forName .getClassloader() seems circular to me. > it would help me immensely if a version of avro-mapred 1.7 for Hadoop 2 could be made available Are code changes required or simply changes in pom.xml dependencies?
          Hide
          Jacob Metcalf added a comment -

          Doug, thanks for doing this. Am based in the UK so tested this morning. There were some issues that meant I had to extend the patch a bit and I am still left with two problems that I have not solved.

          The first task I had was to make the changes to compile avro-mapred against Hadoop 2. These are relatively easy as the incompatiblity is confined to TaskAttemptContext & SequenceFileBase.

          Secondly your patched AvroSerialization whilst it uses the config object is still trying to use the parent ClassLoader so cannot find my Avro class. I have included my attempt to address this in the patch. Basically I locate the classloader using:

          Class.forName( schema.getFullName()).getClassLoader()

          With some additional logic to support UNIONs. I am sure it could be a lot more elegant/efficient but it worked.

          This got me a long way forward. Now Hadoop 2 / Avro 1.7 are able to deserialize my Avro Specific classes in the shuffle. However I am still left with two corollary classloader problems:

          2) Doing a deepCopy via <MyClass>.newBuilder( <myObject> ).build(). Here the problem boils down to another use of SpecificData.get() this time in SpecificRecordBuilderBase.java:

          protected SpecificRecordBuilderBase(Schema schema) {{
          super(schema, SpecificData.get());
          }}

          3) Using AvroKeyValueInputFormat to deserialize from a file. This is another case of needing to pass a SpecificData object to a reader, this time in AvroRecordReaderBase.java:90:

          // Wrap the seekable input stream in an Avro DataFileReader.
          mAvroFileReader = createAvroFileReader(seekableFileInput,
          new ReflectDatumReader<T>(mReaderSchema));

          In terms of taking this forward - Its a big ask but it would help me immensely if a version of avro-mapred 1.7 for Hadoop 2 could be made available. For example the MRUnit team have come up with a way of distributing both Hadoop 1 and 2 versions with users selecting using a <classifier>hadoop2</classifier>. Then, if you are happy with my fix for the first problem, I can help come up with solutions/test fixes for problems 2 & 3. Happy to raise additional JIRAs for all these points.

          Show
          Jacob Metcalf added a comment - Doug, thanks for doing this. Am based in the UK so tested this morning. There were some issues that meant I had to extend the patch a bit and I am still left with two problems that I have not solved. The first task I had was to make the changes to compile avro-mapred against Hadoop 2. These are relatively easy as the incompatiblity is confined to TaskAttemptContext & SequenceFileBase. Secondly your patched AvroSerialization whilst it uses the config object is still trying to use the parent ClassLoader so cannot find my Avro class. I have included my attempt to address this in the patch. Basically I locate the classloader using: Class.forName( schema.getFullName()).getClassLoader() With some additional logic to support UNIONs. I am sure it could be a lot more elegant/efficient but it worked. This got me a long way forward. Now Hadoop 2 / Avro 1.7 are able to deserialize my Avro Specific classes in the shuffle. However I am still left with two corollary classloader problems: 2) Doing a deepCopy via <MyClass>.newBuilder( <myObject> ).build(). Here the problem boils down to another use of SpecificData.get() this time in SpecificRecordBuilderBase.java: protected SpecificRecordBuilderBase(Schema schema) {{ super(schema, SpecificData.get()); }} 3) Using AvroKeyValueInputFormat to deserialize from a file. This is another case of needing to pass a SpecificData object to a reader, this time in AvroRecordReaderBase.java:90: // Wrap the seekable input stream in an Avro DataFileReader. mAvroFileReader = createAvroFileReader(seekableFileInput, new ReflectDatumReader<T>(mReaderSchema)); In terms of taking this forward - Its a big ask but it would help me immensely if a version of avro-mapred 1.7 for Hadoop 2 could be made available. For example the MRUnit team have come up with a way of distributing both Hadoop 1 and 2 versions with users selecting using a <classifier>hadoop2</classifier>. Then, if you are happy with my fix for the first problem, I can help come up with solutions/test fixes for problems 2 & 3. Happy to raise additional JIRAs for all these points.
          Jacob Metcalf made changes -
          Attachment AVRO-1103-for 0.23.1.patch [ 12532297 ]
          Doug Cutting made changes -
          Attachment AVRO-1103.patch [ 12532262 ]
          Hide
          Doug Cutting added a comment -

          New version of patch that adds a test.

          I'll commit this soon unless there are objections.

          Show
          Doug Cutting added a comment - New version of patch that adds a test. I'll commit this soon unless there are objections.
          Doug Cutting made changes -
          Attachment AVRO-1103.patch [ 12532245 ]
          Hide
          Doug Cutting added a comment -

          Improved patch that gets classloader from configuration like other serializers.

          Show
          Doug Cutting added a comment - Improved patch that gets classloader from configuration like other serializers.
          Doug Cutting made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Doug Cutting [ cutting ]
          Fix Version/s 1.7.1 [ 12321552 ]
          Doug Cutting made changes -
          Attachment AVRO-1103.patch [ 12532225 ]
          Hide
          Doug Cutting added a comment -

          Here's a patch that implements what I believe you are suggesting. Existing tests pass.

          Can you please verify whether this works for you? Thanks!

          Show
          Doug Cutting added a comment - Here's a patch that implements what I believe you are suggesting. Existing tests pass. Can you please verify whether this works for you? Thanks!
          Jacob Metcalf made changes -
          Description Continuing on from AVRO-873 I believe some more work needs to be done to get the MapReduce 2 APIs in Avro 1.7 working with Hadoop 0.23. Since it revolves around classloaders it is complex to present a unit test which fails so I will explain the problem:

          - By default SpecificDatumReader will use the classloader it was loaded from to find a Specific class to deserialize into.

          - In earlier versions of Hadoop e.g. 0.20.2 Avro was not included so typically you would bundle Avro into your job jar along with the Specific classes so they would be on the same classpath.
           
          - However later versions of Hadoop such as 0.23 ship with Avro. Thus you find that the SpecificData.class.getClassloader() is typically a parent loader which just contains Hadoop components.

          - Thus when SpecificData goes to construct a Specific class from the schema it cannot locate it and silently defaults to creating a GenericData.

          In AVRO-873 an additional constructor was added to SpecificData to force it to use a different classloader. Thus to extend this fix to the new MR2 APIs:

          - AvroDeserializer could attempt to instantiate the class using Class.forName() and from this get the appropriate Classloader and pass this into the constructor of SpecificDatumReader.

          - Line 51 of SpecificData.java is:

          bq. protected SpecificData() { this(SpecificData.class.getClassLoader()); }

          - This would need to be changed to use this.classloader.

          I have raised this in the mail groups here: http://search-hadoop.com/m/wVUf1aLCwd/classloader/v=threaded so apologies if this is already being thought about.
          Continuing on from AVRO-873 I believe some more work needs to be done to get the MapReduce 2 APIs in Avro 1.7 working with Hadoop 0.23. Since it revolves around classloaders it is complex to present a unit test which fails so I will explain the problem:

          - By default SpecificDatumReader will use the classloader it was loaded from to find a Specific class to deserialize into.

          - In earlier versions of Hadoop e.g. 0.20.2 Avro was not included so typically you would bundle Avro into your job jar along with the Specific classes so they would be on the same classpath.
           
          - However later versions of Hadoop such as 0.23 ship with Avro. Thus you find that the SpecificData.class.getClassloader() is typically a parent loader which just contains Hadoop components.

          - Thus when SpecificData goes to construct a Specific class from the schema it cannot locate it and silently defaults to creating a GenericData.

          In AVRO-873 an additional constructor was added to SpecificData to force it to use a different classloader. Thus to extend this fix to the new MR2 APIs:

          - AvroDeserializer could attempt to instantiate the class using Class.forName() and from this get the appropriate Classloader and pass this into the constructor of SpecificDatumReader.

          - Line 2771 of SpecificData.java is:

          bq. Class c = SpecificData.get().getClass(schema);

          - This would need to be changed to:

          bq. Class c = this.getClass(schema);

          I have raised this in the mail groups here: http://search-hadoop.com/m/wVUf1aLCwd/classloader/v=threaded so apologies if this is already being thought about.
          Jacob Metcalf made changes -
          Field Original Value New Value
          Link This issue is related to AVRO-873 [ AVRO-873 ]
          Jacob Metcalf created issue -

            People

            • Assignee:
              Doug Cutting
              Reporter:
              Jacob Metcalf
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development