Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-964

ClassNotFoundException in ReduceTaskRunner

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 0.11.0
    • None
    • None
    • windows xp and fedora core 6 linux, java 1.5.10...should affect all systems

    Description

      In the ReduceTaskRunner constructor lin 339 a sorter is created that attempts to get the map output key and value classes from the configuration object. This is before the TaskTracker$Child process is spawned off into into own separate JVM so here the classpath for the configuration is the classpath that started the TaskTracker. The current hadoop script includes the hadoop jars, meaning that any hadoop writable type will be found, but it doesn't include nutch jars so any nutch writable type or any other writable type will not be found and will throw a ClassNotFoundException.

      I don't think it is a good idea to have a dependecy on specific Nutch jars in the Hadoop script but it is a good idea to allow jars to be included if they are in specific locations, such as the HADOOP_HOME where the nutch jar resides. I have attached a patch that adds any jars in the HADOOP_HOME directory to the hadoop classpath. This fixes the issues with getting ClassNotFoundExceptions inside of Nutch processes.

      Attachments

        1. classpath.patch
          0.5 kB
          Dennis Kubes
        2. classpath2.path
          2 kB
          Dennis Kubes
        3. classpath2_unittest.patch
          7 kB
          Dennis Kubes
        4. classpath3.patch
          9 kB
          Dennis Kubes

        Issue Links

          Activity

            People

              Unassigned Unassigned
              musepwizard Dennis Kubes
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: