Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running wordcount with trunk gives java.lang.NoClassDefFoundError: org/apache/avro/io/DatumReader. ivy settings need to be updated to get avro jars as well

      1. MR-859.patch
        0.6 kB
        Ravi Gummadi

        Issue Links

          Activity

          Hide
          Jothi Padmanabhan added a comment -

          Stack trace:

          jothipn@doorwheel-lm $ bin/hadoop jar build/hadoop-mapred-examples-0.21.0-dev.jar wordcount input output
          09/08/13 13:58:48 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
          09/08/13 13:58:48 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead.
          09/08/13 13:58:49 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead.
          09/08/13 13:58:49 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead.
          09/08/13 13:58:49 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead.
          09/08/13 13:58:49 INFO input.FileInputFormat: Total input paths to process : 1
          java.lang.NoClassDefFoundError: org/apache/avro/io/DatumReader
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:247)
          at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:837)
          at org.apache.hadoop.io.serializer.SerializationFactory.add(SerializationFactory.java:66)
          at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58)
          at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:925)
          at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:806)
          at org.apache.hadoop.mapreduce.Job.submit(Job.java:489)
          at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:504)
          at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
          at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:140)
          at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:71)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
          Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader
          at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:319)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:330)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:254)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:402)
          ... 22 more

          Show
          Jothi Padmanabhan added a comment - Stack trace: jothipn@doorwheel-lm $ bin/hadoop jar build/hadoop-mapred-examples-0.21.0-dev.jar wordcount input output 09/08/13 13:58:48 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively 09/08/13 13:58:48 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead. 09/08/13 13:58:49 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead. 09/08/13 13:58:49 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead. 09/08/13 13:58:49 WARN fs.FileSystem: "localhost:7597" is a deprecated filesystem name. Use "hdfs://localhost:7597/" instead. 09/08/13 13:58:49 INFO input.FileInputFormat: Total input paths to process : 1 java.lang.NoClassDefFoundError: org/apache/avro/io/DatumReader at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:837) at org.apache.hadoop.io.serializer.SerializationFactory.add(SerializationFactory.java:66) at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:925) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:806) at org.apache.hadoop.mapreduce.Job.submit(Job.java:489) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:504) at org.apache.hadoop.examples.WordCount.main(WordCount.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:140) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:319) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:330) at java.lang.ClassLoader.loadClass(ClassLoader.java:254) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:402) ... 22 more
          Hide
          Ravi Gummadi added a comment -

          The issue happens only if we copy hadoop-core-0.21.0-dev.jar from common/trunk/build by building it.

          Attaching patch that makes necessary changes to ivy.xml.

          Show
          Ravi Gummadi added a comment - The issue happens only if we copy hadoop-core-0.21.0-dev.jar from common/trunk/build by building it. Attaching patch that makes necessary changes to ivy.xml.
          Hide
          Jothi Padmanabhan added a comment -

          +1. I am able to run word count after applying the patch and doing ant

          Show
          Jothi Padmanabhan added a comment - +1. I am able to run word count after applying the patch and doing ant
          Hide
          Sharad Agarwal added a comment -

          I just committed this. Thanks Ravi.

          Show
          Sharad Agarwal added a comment - I just committed this. Thanks Ravi.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi, just want to let you know that the versions rev is better to be defined with variables specified in ivy/libraries.properties. I learned this from Tom in HDFS-534 earlier.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi, just want to let you know that the versions rev is better to be defined with variables specified in ivy/libraries.properties. I learned this from Tom in HDFS-534 earlier.
          Hide
          Ravi Gummadi added a comment -

          OK. That looks better.
          I copied this from common/trunk/ivy.xml. So both common/trunk/ivy.xml and mapreduce/trunk/ivy.xml can be modified to have the rev variable in ivy/libraries.properties in some patches later.

          Show
          Ravi Gummadi added a comment - OK. That looks better. I copied this from common/trunk/ivy.xml. So both common/trunk/ivy.xml and mapreduce/trunk/ivy.xml can be modified to have the rev variable in ivy/libraries.properties in some patches later.

            People

            • Assignee:
              Ravi Gummadi
              Reporter:
              Jothi Padmanabhan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development