Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5831

Old MR client is not compatible with new MR application

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 2.2.0, 2.3.0
    • 2.6.0
    • client, mr-am
    • None
    • Reviewed

    Description

      Recently, we saw the following scenario:

      1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR 2.3.

      2. The user client on a machine that MR 2.2 is installed and in the classpath.

      Then, when the user submitted a simple wordcount job, he saw the following message:

      16:00:41,027  INFO main mapreduce.Job:1345 -  map 100% reduce 100%
      16:00:41,036  INFO main mapreduce.Job:1356 - Job job_1396468045458_0006 completed successfully
      16:02:20,535  WARN main mapreduce.JobRunner:212 - Cannot start job [wordcountJob]
      java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
      	at java.lang.Enum.valueOf(Enum.java:236)
      	at org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
      	at org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
      	at org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
      	at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
      	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
      	at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
      	at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
      	at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
      	at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
      	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
      	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
              . . .
      

      The problem is that the wordcount job was running on one or more than one nodes of the YARN cluster, where MR 2.3 libs were installed, and JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side, due to the classpath setting, the client was likely to run with MR 2.2 libs. After the client retrieved the counters from MR AM, it tried to construct the Counter object with the received counter name. Unfortunately, the enum didn't exist in the client's classpath. Therefore, "No enum constant" exception is thrown here.

      JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since Hadoop 2.3.

      Attachments

        1. MAPREDUCE-5831.patch
          6 kB
          Junping Du
        2. MAPREDUCE-5831-v2.patch
          8 kB
          Junping Du
        3. MAPREDUCE-5831-v3.patch
          9 kB
          Junping Du

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            junping_du Junping Du
            zjshen Zhijie Shen
            Votes:
            1 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment