Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.2.0
    • Component/s: examples, mrv1
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Using branch-1 tarball, if the user tries to submit an example aggregatewordcount, the job fails with the following error:

      ahmed@ubuntu:~/demo/deploy/hadoop-1.2.0-SNAPSHOT$ bin/hadoop jar hadoop-examples-1.2.0-SNAPSHOT.jar aggregatewordcount input examples-output/aggregatewordcount 2 textinputformat
      12/09/12 17:09:46 INFO mapred.JobClient: originalJarPath: /home/ahmed/demo/deploy/hadoop-1.2.0-SNAPSHOT/hadoop-core-1.2.0-SNAPSHOT.jar
      12/09/12 17:09:48 INFO mapred.JobClient: submitJarFile: hdfs://localhost:9000/tmp/hadoop-ahmed/mapred/staging/ahmed/.staging/job_201209121702_0008/job.jar
      12/09/12 17:09:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      12/09/12 17:09:48 WARN snappy.LoadSnappy: Snappy native library not loaded
      12/09/12 17:09:48 INFO mapred.FileInputFormat: Total input paths to process : 21
      12/09/12 17:09:49 INFO mapred.JobClient: Running job: job_201209121702_0008
      12/09/12 17:09:50 INFO mapred.JobClient:  map 0% reduce 0%
      12/09/12 17:09:58 INFO mapred.JobClient: Task Id : attempt_201209121702_0008_m_000000_0, Status : FAILED
      java.lang.RuntimeException: Error in configuring object
      	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
      	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
      	at org.apache.hadoop.mapred.Child.main(Child.java:249)
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
      	... 9 more
      Caused by: java.lang.RuntimeException: Error in configuring object
      	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
      	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
      	at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
      	... 14 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
      	... 17 more
      Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.examples.AggregateWordCount$WordCountPlugInClass
      	at org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance(UserDefinedValueAggregatorDescriptor.java:57)
      	at org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createAggregator(UserDefinedValueAggregatorDescriptor.java:64)
      	at org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.<init>(UserDefinedValueAggregatorDescriptor.java:76)
      	at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggregatorDescriptor(ValueAggregatorJobBase.java:54)
      	at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregatorDescriptors(ValueAggregatorJobBase.java:65)
      	at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMySpec(ValueAggregatorJobBase.java:74)
      	at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(ValueAggregatorJobBase.java:42)
      	... 22 more
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.examples.AggregateWordCount$WordCountPlugInClass
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:247)
      	at org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance(UserDefinedValueAggregatorDescriptor.java:52)
      	... 28 more
      
      1. MAPREDUCE-4652_rev2.patch
        5 kB
        Ahmed Radwan
      2. MAPREDUCE-4652.patch
        4 kB
        Ahmed Radwan

        Activity

        Hide
        Ahmed Radwan added a comment -

        The issue here is that hadoop-examples.jar is missing from the task's classpath since ValueAggregatorJob sets its own jar as the job jar, so hadoop-core.jar is used instead of hadoop-examples.jar. A workaround could be to explicitly add hadoop-examples.jar to the task's classpath via "-libjars" argument or the "user.jar.file" property. I'll be uploading a patch momentarily which fixes this default behavior of ValueAggregatorJob.

        Show
        Ahmed Radwan added a comment - The issue here is that hadoop-examples.jar is missing from the task's classpath since ValueAggregatorJob sets its own jar as the job jar, so hadoop-core.jar is used instead of hadoop-examples.jar. A workaround could be to explicitly add hadoop-examples.jar to the task's classpath via "-libjars" argument or the "user.jar.file" property. I'll be uploading a patch momentarily which fixes this default behavior of ValueAggregatorJob.
        Hide
        Ahmed Radwan added a comment -

        Here is a patch that modifies the ValueAggregatorJob behavior to use the caller's jar instead of its own jar. So for the example mentioned above, the hadoop-examples.jar is set as the job jar.

        I have manually tested this fix on a single node cluster and confirmed that the example runs fine.

        Show
        Ahmed Radwan added a comment - Here is a patch that modifies the ValueAggregatorJob behavior to use the caller's jar instead of its own jar. So for the example mentioned above, the hadoop-examples.jar is set as the job jar. I have manually tested this fix on a single node cluster and confirmed that the example runs fine.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544911/MAPREDUCE-4652.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2848//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544911/MAPREDUCE-4652.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2848//console This message is automatically generated.
        Hide
        Tom White added a comment -

        For compatibility we should keep the existing createValueAggregatorJob() methods on ValueAggregatorJob - so add new overloaded variants rather than change the existing ones. Otherwise looks good.

        Show
        Tom White added a comment - For compatibility we should keep the existing createValueAggregatorJob() methods on ValueAggregatorJob - so add new overloaded variants rather than change the existing ones. Otherwise looks good.
        Hide
        Ahmed Radwan added a comment -

        Many thanks Tom for the review! Here is the updated version addressing your comments.

        Show
        Ahmed Radwan added a comment - Many thanks Tom for the review! Here is the updated version addressing your comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12545623/MAPREDUCE-4652_rev2.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2862//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545623/MAPREDUCE-4652_rev2.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2862//console This message is automatically generated.
        Hide
        Ahmed Radwan added a comment -

        I have locally ran test-patch and here is the result:

        
             [exec] BUILD SUCCESSFUL
             [exec] Total time: 5 minutes 49 seconds
             [exec] 
             [exec] 
             [exec] 
             [exec] 
             [exec] -1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
             [exec]                         Please justify why no tests are needed for this patch.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     -1 findbugs.  The patch appears to introduce 221 new Findbugs (version 2.0.1) warnings.
        

        I have inspected the findbugs report and the warnings seem unrelated to the current patch. The number 221 seems to come from the total number of warnings in the report 432 minus the OK_FINDBUGS_WARNINGS=211 in src/test/test-patch.properties, so it doesn't seem to reflect a real number of new warnings introduced by the patch.

        Show
        Ahmed Radwan added a comment - I have locally ran test-patch and here is the result: [exec] BUILD SUCCESSFUL [exec] Total time: 5 minutes 49 seconds [exec] [exec] [exec] [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 221 new Findbugs (version 2.0.1) warnings. I have inspected the findbugs report and the warnings seem unrelated to the current patch. The number 221 seems to come from the total number of warnings in the report 432 minus the OK_FINDBUGS_WARNINGS=211 in src/test/test-patch.properties, so it doesn't seem to reflect a real number of new warnings introduced by the patch.
        Hide
        Tom White added a comment -

        I just committed this. Thanks Ahmed!

        Show
        Tom White added a comment - I just committed this. Thanks Ahmed!
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop 1.2.0.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development