Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5957

AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      With the job classloader enabled, the MR AM throws ClassNotFoundException if a custom output format class is specified.

      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not found
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
      Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not found
      	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
      	at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
      	... 8 more
      Caused by: java.lang.ClassNotFoundException: Class com.foo.test.TestOutputFormat not found
      	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
      	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
      	... 10 more
      
      1. MAPREDUCE-5957.branch-2.patch
        26 kB
        Jason Lowe
      2. MAPREDUCE-5957.patch
        28 kB
        Sangjin Lee
      3. MAPREDUCE-5957.patch
        28 kB
        Sangjin Lee
      4. MAPREDUCE-5957.patch
        26 kB
        Sangjin Lee
      5. MAPREDUCE-5957.patch
        20 kB
        Sangjin Lee
      6. MAPREDUCE-5957.patch
        13 kB
        Sangjin Lee
      7. MAPREDUCE-5957.patch
        6 kB
        Sangjin Lee

        Issue Links

          Activity

            People

            • Assignee:
              Sangjin Lee
              Reporter:
              Sangjin Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development