Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12406

AbstractMapWritable.readFields throws ClassNotFoundException with custom writables

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: io
    • Labels:
      None
    • Environment:

      Ubuntu Linux 14.04 LTS amd64

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Flags:
      Patch

      Description

      Note: I am not an expert at JAVA, Class loaders, or Hadoop. I am just a hacker. My solution might be entirely wrong.

      AbstractMapWritable.readFields throws a ClassNotFoundException when reading custom writables. Debugging the job using remote debugging in IntelliJ revealed that the class loader being used in Class.forName() is different than that used by the Thread's current context (Thread.currentThread().getContextClassLoader()). The class path for the system class loader does not include the libraries of the job jar. However, the class path for the context class loader does. The proposed patch changes the class loading mechanism in readFields to use the Thread's context class loader instead of the system's default class loader.

      1. HADOOP-12406.1.patch
        2 kB
        Vinod Kumar Vavilapalli
      2. HADOOP-12406.patch
        1 kB
        Nadeem Douba

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755546/abstractmapwritable.patch
          Optional Tests  
          git revision trunk / d845547
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7648/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755546/abstractmapwritable.patch Optional Tests   git revision trunk / d845547 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7648/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 12s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 56s There were no new javac warning messages.
          +1 javadoc 9m 54s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 7s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 3s Tests failed in hadoop-common.
              62m 30s  



          Reason Tests
          Failed unit tests hadoop.ipc.TestSaslRPC



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755547/HADOOP-12406.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d845547
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7649/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7649/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7649/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 12s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 7s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 3s Tests failed in hadoop-common.     62m 30s   Reason Tests Failed unit tests hadoop.ipc.TestSaslRPC Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755547/HADOOP-12406.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d845547 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7649/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7649/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7649/console This message was automatically generated.
          Hide
          chrismattmann Chris A. Mattmann added a comment -

          Hey Everyone:

          This is related to a problem we are seeing in NUTCH, NUTCH-1084. Any chance someone could review. It looks like the test failures are actually orthogonal to this patch.

          Show
          chrismattmann Chris A. Mattmann added a comment - Hey Everyone: This is related to a problem we are seeing in NUTCH, NUTCH-1084 . Any chance someone could review. It looks like the test failures are actually orthogonal to this patch.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Nadeem Douba, please use the "Target Version" field to express your intention. Fix-version is exclusively used by committers when a patch gets committed.

          IAC, 2.7.1 is done, targeting this for 2.7.2/2.8.0.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Nadeem Douba , please use the "Target Version" field to express your intention. Fix-version is exclusively used by committers when a patch gets committed. IAC, 2.7.1 is done, targeting this for 2.7.2/2.8.0.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Nadeem Douba, is this happening inside a MapReduce job running on top of YARN? MapReduce does have job.jar in the system classpath, so that's not explainable.

          The only way this may happen is if you are using the MapReduce JobClassLoader for this - please let us know the values for mapreduce.job.classloader in your job.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Nadeem Douba , is this happening inside a MapReduce job running on top of YARN? MapReduce does have job.jar in the system classpath, so that's not explainable. The only way this may happen is if you are using the MapReduce JobClassLoader for this - please let us know the values for mapreduce.job.classloader in your job.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Moving this out of 2.7.2 as there's been no update in a while.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Moving this out of 2.7.2 as there's been no update in a while.
          Hide
          ndouba Nadeem Douba added a comment -

          Hi Vinod,

          I was running this on top of YARN with the default configuration enabled on top of Java 1.8 from Oracle. I think the new isolation strategy used in the newer versions of the JVM for class loaders prevents the system class loader from loading a job jar class. I didn't do anything fancy with configuration options or anything else. The only thing I added were the remote debugging options to allow me to attach to the map and reduce threads remotely and debug them step by step. I can tell you that the job jar definitely isn't in the system's class loader but in the current thread's context class loader. I verified this observation by manually inspecting the class loader object from the system and the one that comes with the thread context.

          Cheers,

          Nadeem

          Show
          ndouba Nadeem Douba added a comment - Hi Vinod, I was running this on top of YARN with the default configuration enabled on top of Java 1.8 from Oracle. I think the new isolation strategy used in the newer versions of the JVM for class loaders prevents the system class loader from loading a job jar class. I didn't do anything fancy with configuration options or anything else. The only thing I added were the remote debugging options to allow me to attach to the map and reduce threads remotely and debug them step by step. I can tell you that the job jar definitely isn't in the system's class loader but in the current thread's context class loader. I verified this observation by manually inspecting the class loader object from the system and the one that comes with the thread context. Cheers, Nadeem
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Hi Nadeem Douba,

          I'm about to do a 2.7.3 Apache Hadoop release and finally got around to this again.

          Analysis

          To make progress, I had to read up a bit on nutch and about how to run this so that I can reproduce the bug in order to rationalize your patch. I finally succeeded in doing so! Tested this with 2.7.2 release and nutch 1.11 and using the URL feed given at NUTCH-1084

          ~/tmp/common/hadoop-common-2.7.2/bin/hadoop jar apache-nutch-1.11.job org.apache.nutch.crawl.CrawlDbReader file:///tmp/nutch/apache-nutch-1.11/runtime/local/crawl/crawldb/ -url http://bappenas.go.id/
          

          I can reproduce all the problems listed at NUTCH-1084 - with readdb, MR local-job-runner based job for crawling etc.

          The real issue is that Nutch's readdb is client-only and not running a MapReduce job which was my question before. For regular MR jobs, the job-jar is on the system class-loader. For the client-only invocations using "hadoop jar" and local-job-runner, the job-jar is actually not on the system-classpath - that is why you are running into the issue.

          Summary

          Your patch looks good to me. Clearly, the thread context-loader falls back to system class-loader where it is not overridden - so we are fine for all the ways of loading the classes in readFields.

          I'll resubmit your patch with minor commenting related changes to Jenkins and commit if Mr.Jenkins is also fine.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Hi Nadeem Douba , I'm about to do a 2.7.3 Apache Hadoop release and finally got around to this again. Analysis To make progress, I had to read up a bit on nutch and about how to run this so that I can reproduce the bug in order to rationalize your patch. I finally succeeded in doing so! Tested this with 2.7.2 release and nutch 1.11 and using the URL feed given at NUTCH-1084 ~/tmp/common/hadoop-common-2.7.2/bin/hadoop jar apache-nutch-1.11.job org.apache.nutch.crawl.CrawlDbReader file: ///tmp/nutch/apache-nutch-1.11/runtime/local/crawl/crawldb/ -url http://bappenas.go.id/ I can reproduce all the problems listed at NUTCH-1084 - with readdb, MR local-job-runner based job for crawling etc. The real issue is that Nutch's readdb is client-only and not running a MapReduce job which was my question before. For regular MR jobs, the job-jar is on the system class-loader. For the client-only invocations using "hadoop jar" and local-job-runner, the job-jar is actually not on the system-classpath - that is why you are running into the issue. Summary Your patch looks good to me. Clearly, the thread context-loader falls back to system class-loader where it is not overridden - so we are fine for all the ways of loading the classes in readFields. I'll resubmit your patch with minor commenting related changes to Jenkins and commit if Mr.Jenkins is also fine.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Updated patch fixing the code comment..

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Updated patch fixing the code comment..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 25s trunk passed
          +1 compile 5m 39s trunk passed with JDK v1.8.0_77
          +1 compile 6m 32s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 0m 53s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 41s the patch passed
          +1 compile 5m 35s the patch passed with JDK v1.8.0_77
          +1 javac 5m 35s the patch passed
          +1 compile 6m 32s the patch passed with JDK v1.7.0_95
          +1 javac 6m 32s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 3s the patch passed with JDK v1.7.0_95
          -1 unit 7m 21s hadoop-common in the patch failed with JDK v1.8.0_77.
          +1 unit 7m 57s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          58m 34s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797942/HADOOP-12406.1.patch
          JIRA Issue HADOOP-12406
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b4a0f8dc2758 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1b78b2b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 25s trunk passed +1 compile 5m 39s trunk passed with JDK v1.8.0_77 +1 compile 6m 32s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 53s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 41s the patch passed +1 compile 5m 35s the patch passed with JDK v1.8.0_77 +1 javac 5m 35s the patch passed +1 compile 6m 32s the patch passed with JDK v1.7.0_95 +1 javac 6m 32s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 3s the patch passed with JDK v1.7.0_95 -1 unit 7m 21s hadoop-common in the patch failed with JDK v1.8.0_77. +1 unit 7m 57s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 58m 34s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797942/HADOOP-12406.1.patch JIRA Issue HADOOP-12406 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b4a0f8dc2758 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1b78b2b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9055/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          markus17 Markus Jelsma added a comment -

          Hi Vinod and Nadeem, thanks for taking care of this. At Apache Nutch, we're looking forward to 2.7.3!

          Show
          markus17 Markus Jelsma added a comment - Hi Vinod and Nadeem, thanks for taking care of this. At Apache Nutch, we're looking forward to 2.7.3!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          TestReloadingX509TrustManager failure is not related, I'll see if there is an existing JIRA.

          Checking this in now..

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - TestReloadingX509TrustManager failure is not related, I'll see if there is an existing JIRA. Checking this in now..
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Committed this to trunk, branch-2 and branch-2.7. Thanks Nadeem Douba!

          Forgot to mention that I've tested that the nutch code fails without the patch and passes with.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Committed this to trunk, branch-2 and branch-2.7. Thanks Nadeem Douba ! Forgot to mention that I've tested that the nutch code fails without the patch and passes with.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9593 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9593/)
          HADOOP-12406. Fixed AbstractMapWritable.readFields to use the thread's (vinodkv: rev 069c6c62def4a0f94382e9f149581d8e22f6d31c)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/AbstractMapWritable.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9593 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9593/ ) HADOOP-12406 . Fixed AbstractMapWritable.readFields to use the thread's (vinodkv: rev 069c6c62def4a0f94382e9f149581d8e22f6d31c) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/AbstractMapWritable.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          djp Junping Du added a comment -

          Merge the fix to branch-2.8.

          Show
          djp Junping Du added a comment - Merge the fix to branch-2.8.

            People

            • Assignee:
              ndouba Nadeem Douba
              Reporter:
              ndouba Nadeem Douba
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development