Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5831

Old MR client is not compatible with new MR application

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 2.3.0
    • Fix Version/s: 2.6.0
    • Component/s: client, mr-am
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Recently, we saw the following scenario:

      1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR 2.3.

      2. The user client on a machine that MR 2.2 is installed and in the classpath.

      Then, when the user submitted a simple wordcount job, he saw the following message:

      16:00:41,027  INFO main mapreduce.Job:1345 -  map 100% reduce 100%
      16:00:41,036  INFO main mapreduce.Job:1356 - Job job_1396468045458_0006 completed successfully
      16:02:20,535  WARN main mapreduce.JobRunner:212 - Cannot start job [wordcountJob]
      java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
      	at java.lang.Enum.valueOf(Enum.java:236)
      	at org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
      	at org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
      	at org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
      	at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
      	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
      	at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
      	at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
      	at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
      	at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
      	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
      	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
              . . .
      

      The problem is that the wordcount job was running on one or more than one nodes of the YARN cluster, where MR 2.3 libs were installed, and JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side, due to the classpath setting, the client was likely to run with MR 2.2 libs. After the client retrieved the counters from MR AM, it tried to construct the Counter object with the received counter name. Unfortunately, the enum didn't exist in the client's classpath. Therefore, "No enum constant" exception is thrown here.

      JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since Hadoop 2.3.

      1. MAPREDUCE-5831-v3.patch
        9 kB
        Junping Du
      2. MAPREDUCE-5831-v2.patch
        8 kB
        Junping Du
      3. MAPREDUCE-5831.patch
        6 kB
        Junping Du

        Issue Links

          Activity

          Hide
          Junping Du added a comment -

          Hi Zhijie Shen, I think problem get fixed since 2.6. For 2.2, because it doesn't have any fix, the problem could be still there. Please see my comments in MAPREDUCE-6168.

          Show
          Junping Du added a comment - Hi Zhijie Shen , I think problem get fixed since 2.6. For 2.2, because it doesn't have any fix, the problem could be still there. Please see my comments in MAPREDUCE-6168 .
          Hide
          Zhijie Shen added a comment -

          The problem is not completely resolved. File another jira for it: MAPREDUCE-6168.

          Show
          Zhijie Shen added a comment - The problem is not completely resolved. File another jira for it: MAPREDUCE-6168 .
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1908 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1908/)
          MAPREDUCE-5831. Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1908 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1908/ ) MAPREDUCE-5831 . Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1883 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1883/)
          MAPREDUCE-5831. Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1883 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1883/ ) MAPREDUCE-5831 . Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #692 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/692/)
          MAPREDUCE-5831. Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #692 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/692/ ) MAPREDUCE-5831 . Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6118 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6118/)
          MAPREDUCE-5831. Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6118 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6118/ ) MAPREDUCE-5831 . Make MR client ignore unknown counters received from AM. Contributed by Junping Du. (zjshen: rev 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          Hide
          Zhijie Shen added a comment -

          Committed to trunk and branch-2. Thanks, Junping!

          Show
          Zhijie Shen added a comment - Committed to trunk and branch-2. Thanks, Junping!
          Hide
          Zhijie Shen added a comment -

          +1, will commit it

          Show
          Zhijie Shen added a comment - +1, will commit it
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671374/MAPREDUCE-5831-v3.patch
          against trunk revision e96ce6f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4917//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4917//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671374/MAPREDUCE-5831-v3.patch against trunk revision e96ce6f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4917//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4917//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks Zhijie Shen for review and comments. v3 patch just addressed your comments.

          Show
          Junping Du added a comment - Thanks Zhijie Shen for review and comments. v3 patch just addressed your comments.
          Hide
          Zhijie Shen added a comment -

          Good to me in general, but for FileSystemCounterGroup, it seems that we need to:
          1. fix addCounter as well as we did for FrameworkCounterGroup, and
          2. add an analog test case for FileSystemCounterGroup.

          /home/jenkins/tools/maven/latest/bin/mvn eclipse:eclipse -DHadoopPatchProcess > /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/../patchprocess/patchEclipseOutput.txt 2>&1
          /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/dev-support/test-patch.sh: line 693: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/../patchprocess/patchEclipseOutput.txt: No such file or directory
          

          it seems that the build corrupted for some reason.

          Show
          Zhijie Shen added a comment - Good to me in general, but for FileSystemCounterGroup, it seems that we need to: 1. fix addCounter as well as we did for FrameworkCounterGroup, and 2. add an analog test case for FileSystemCounterGroup. /home/jenkins/tools/maven/latest/bin/mvn eclipse:eclipse -DHadoopPatchProcess > /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/../patchprocess/patchEclipseOutput.txt 2>&1 /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/dev-support/test-patch.sh: line 693: /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/../patchprocess/patchEclipseOutput.txt: No such file or directory it seems that the build corrupted for some reason.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671342/MAPREDUCE-5831-v2.patch
          against trunk revision 8269bfa.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4916//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4916//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671342/MAPREDUCE-5831-v2.patch against trunk revision 8269bfa. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 eclipse:eclipse . The patch failed to build with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4916//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4916//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          It's fine too. If so, we need to fix FileSystemCounterGroup.findCounter, which will have the similar issue.

          Good point. Address this in v2 version.

          Show
          Junping Du added a comment - It's fine too. If so, we need to fix FileSystemCounterGroup.findCounter, which will have the similar issue. Good point. Address this in v2 version.
          Hide
          Zhijie Shen added a comment -

          I prefer to handle it in findCounter because as a find* api, it shouldn't crash with exception if cannot find something. Also, this way is easier to test.

          It's fine too. If so, we need to fix FileSystemCounterGroup.findCounter, which will have the similar issue.

          Show
          Zhijie Shen added a comment - I prefer to handle it in findCounter because as a find* api, it shouldn't crash with exception if cannot find something. Also, this way is easier to test. It's fine too. If so, we need to fix FileSystemCounterGroup.findCounter, which will have the similar issue.
          Hide
          Junping Du added a comment -

          Thanks Zhijie Shen for review and comments.

          For setting value, it seems that we just need to fix TypeConverter, which is the only one used at the interface between client and AM. And the incompatibility happens when client and AM are of different version. Within the scope of either client or AM only, findCounter should have no problem.

          findCounter will throw exception. Just as you said, we can choose to handle this exception in TypeConverter (the caller) or in findCounter. I prefer to handle it in findCounter because as a find* api, it shouldn't crash with exception if cannot find something. Also, this way is easier to test.

          It will not be helpful, because we use the existing Enum object's name to find this object.

          The helpful thing is next "Unknown", it shows it can get a null count instead of throwing exception the NUM_FAILED_MAPS here is just a compare (not null).

          +    org.apache.hadoop.mapreduce.Counter count1 = 
          +        counterGroup.findCounter(JobCounter.NUM_FAILED_MAPS.toString());
          +    Assert.assertNotNull(count1);
          +    
          +    // Verify no exception get thrown when finding an unknown counter
          +    org.apache.hadoop.mapreduce.Counter count2 = 
          +        counterGroup.findCounter("Unknown");
          +    Assert.assertNull(count2);
          
          Show
          Junping Du added a comment - Thanks Zhijie Shen for review and comments. For setting value, it seems that we just need to fix TypeConverter, which is the only one used at the interface between client and AM. And the incompatibility happens when client and AM are of different version. Within the scope of either client or AM only, findCounter should have no problem. findCounter will throw exception. Just as you said, we can choose to handle this exception in TypeConverter (the caller) or in findCounter. I prefer to handle it in findCounter because as a find* api, it shouldn't crash with exception if cannot find something. Also, this way is easier to test. It will not be helpful, because we use the existing Enum object's name to find this object. The helpful thing is next "Unknown", it shows it can get a null count instead of throwing exception the NUM_FAILED_MAPS here is just a compare (not null). + org.apache.hadoop.mapreduce.Counter count1 = + counterGroup.findCounter(JobCounter.NUM_FAILED_MAPS.toString()); + Assert.assertNotNull(count1); + + // Verify no exception get thrown when finding an unknown counter + org.apache.hadoop.mapreduce.Counter count2 = + counterGroup.findCounter( "Unknown" ); + Assert.assertNull(count2);
          Hide
          Zhijie Shen added a comment -

          Junping Du, thanks for the patch. Here are my comments.

          1. For setting value, it seems that we just need to fix TypeConverter, which is the only one used at the interface between client and AM. And the incompatibility happens when client and AM are of different version. Within the scope of either client or AM only, findCounter should have no problem.

          -        c.setValue(yCntr.getValue());
          +        // if c can be found, or it will be skipped.
          +        if (c != null) {
          +          c.setValue(yCntr.getValue());
          +        }
          

          2. The exact problem happens:

            @Override
            public C findCounter(String counterName) {
              return findCounter(valueOf(counterName));
            }
          

          valueOf is going to throw exception because the Enum object doesn't exist at the client side, instead of returning a null here.
          Therefore, in the test:

          +        counterGroup.findCounter(JobCounter.NUM_FAILED_MAPS.toString());
          

          It will not be helpful, because we use the existing Enum object's name to find this object.

          Show
          Zhijie Shen added a comment - Junping Du , thanks for the patch. Here are my comments. 1. For setting value, it seems that we just need to fix TypeConverter, which is the only one used at the interface between client and AM. And the incompatibility happens when client and AM are of different version. Within the scope of either client or AM only, findCounter should have no problem. - c.setValue(yCntr.getValue()); + // if c can be found, or it will be skipped. + if (c != null ) { + c.setValue(yCntr.getValue()); + } 2. The exact problem happens: @Override public C findCounter( String counterName) { return findCounter(valueOf(counterName)); } valueOf is going to throw exception because the Enum object doesn't exist at the client side, instead of returning a null here. Therefore, in the test: + counterGroup.findCounter(JobCounter.NUM_FAILED_MAPS.toString()); It will not be helpful, because we use the existing Enum object's name to find this object.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12670990/MAPREDUCE-5831.patch
          against trunk revision 7af4c38.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4910//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4910//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670990/MAPREDUCE-5831.patch against trunk revision 7af4c38. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4910//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4910//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Upload a patch to fix this issue.

          Show
          Junping Du added a comment - Upload a patch to fix this issue.
          Hide
          Junping Du added a comment -

          Taking it over and mark this as blocker as we will support rolling upgrade since 2.6.

          Show
          Junping Du added a comment - Taking it over and mark this as blocker as we will support rolling upgrade since 2.6.
          Hide
          Wangda Tan added a comment -

          +1 for migrating towards MAPREDUCE-4421 for solution of client/server capability issue in a short time. For the longer time, as mentioned by Vinod, wire capability and rolling upgrade should be the ultimate solution. So IMHO, we don't need fix this issue for now.

          Show
          Wangda Tan added a comment - +1 for migrating towards MAPREDUCE-4421 for solution of client/server capability issue in a short time. For the longer time, as mentioned by Vinod, wire capability and rolling upgrade should be the ultimate solution. So IMHO, we don't need fix this issue for now.
          Hide
          Zhijie Shen added a comment -

          This came up in the context of 2.4.1

          We solve this on our clusters by leveraging MAPREDUCE-442

          and on mailing list:

          Actually, that is a reasonable expectation - particularly because we should all be migrating towards MAPREDUCE-4421 and should stop installing MR on every node.

          Move it out of 2.4.1. Upon the discussion, do you think we even don't want to fix the issue given we are supposed to deploy MR via distributed cache?

          Show
          Zhijie Shen added a comment - This came up in the context of 2.4.1 We solve this on our clusters by leveraging MAPREDUCE-442 and on mailing list: Actually, that is a reasonable expectation - particularly because we should all be migrating towards MAPREDUCE-4421 and should stop installing MR on every node. Move it out of 2.4.1. Upon the discussion, do you think we even don't want to fix the issue given we are supposed to deploy MR via distributed cache?
          Hide
          Jason Lowe added a comment -

          We solve this on our clusters by leveraging MAPREDUCE-4421 (i.e.: deploy MR via HDFS) and have our client configs ask for the same version of the MR framework that they are. That way MR clients always use the same jars for AMs and tasks that match their version, so we won't have an old jobclient trying to talk to a new AM or vice-versa.

          Show
          Jason Lowe added a comment - We solve this on our clusters by leveraging MAPREDUCE-4421 (i.e.: deploy MR via HDFS) and have our client configs ask for the same version of the MR framework that they are. That way MR clients always use the same jars for AMs and tasks that match their version, so we won't have an old jobclient trying to talk to a new AM or vice-versa.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This came up in the context of 2.4.1 Here's what I posted on the mailing lists

          Unfortunately even though we documented wire compatiblity, cross-version client/server support doesn't yet really work for YARN and MapReduce. We can only do that once we have wire-compatibility and eventually rolling upgrades. No fix is in sight for MAPREDUCE-5831 - for now both clients and AMs have to upgrade together in the MapReduce land..

          Show
          Vinod Kumar Vavilapalli added a comment - This came up in the context of 2.4.1 Here's what I posted on the mailing lists Unfortunately even though we documented wire compatiblity, cross-version client/server support doesn't yet really work for YARN and MapReduce. We can only do that once we have wire-compatibility and eventually rolling upgrades. No fix is in sight for MAPREDUCE-5831 - for now both clients and AMs have to upgrade together in the MapReduce land..
          Hide
          Arun C Murthy added a comment -

          I think the real way, going fwd, to fix this is to ensure the MR client and the application are on the same version - MAPREDUCE-4421 should solve that; and be the default deployment model.

          Show
          Arun C Murthy added a comment - I think the real way, going fwd, to fix this is to ensure the MR client and the application are on the same version - MAPREDUCE-4421 should solve that; and be the default deployment model.
          Hide
          Wangda Tan added a comment -

          Link this issue with MAPREDUCE-4150

          Show
          Wangda Tan added a comment - Link this issue with MAPREDUCE-4150

            People

            • Assignee:
              Junping Du
              Reporter:
              Zhijie Shen
            • Votes:
              1 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development