Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9924

FileUtil.createJarWithClassPath() does not generate relative classpath correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 0.23.9
    • Fix Version/s: 2.1.1-beta
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      On Windows, FileUtil.createJarWithClassPath() is called to generate a manifest jar file to pack classpath - to avoid the problem of classpath being too long.
      However, the relative classpath is not handled correctly. It relies on Java's File(relativePath) to resolve the relative path. But it really should be using the given pwd parameter to resolve the relative path.

      To reproduce this bug, you can try some pig job on Windows, it will fail and the pig log on the application master will look like this:

      2013-08-29 23:25:55,498 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat not found
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat not found

      This is because the PigOutputFormat class is in the job.jar file but the classpath manifest has:
      file:/c:/apps/dist/hadoop-2.1.0-beta/bin/job.jar/job.jar
      When it really should be:
      file:/<job container folder>/job.jar/job.jar

        Attachments

        1. HADOOP-9924.patch
          5 kB
          shanyu zhao
        2. HADOOP-9924-2.patch
          6 kB
          shanyu zhao
        3. HADOOP-9924-3.patch
          3 kB
          shanyu zhao
        4. HADOOP-9924-4.patch
          3 kB
          shanyu zhao
        5. HADOOP-9924.addendum.patch
          2 kB
          shanyu zhao

          Activity

            People

            • Assignee:
              shanyu shanyu zhao
              Reporter:
              shanyu shanyu zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: