Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4610

Support deprecated mapreduce.job.counters.limit property in MR2

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The property mapreduce.job.counters.limit was introduced in MAPREDUCE-1943, but the mechanism was changed in MAPREDUCE-901 where the property name was changed to mapreduce.job.counters.max without supporting the old name. We should deprecate but honour the old name to make it easier for folks to move from Hadoop 1 to Hadoop 2.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1h 44m 1 Tom White 30/Aug/12 15:13
        Patch Available Patch Available Resolved Resolved
        3h 3m 1 Tom White 30/Aug/12 18:16
        Resolved Resolved Closed Closed
        42d 32m 1 Arun C Murthy 11/Oct/12 18:48
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #387 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/387/)
        svn merge -c 1379022 FIXES: MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in MR2. (Revision 1390670)

        Result = UNSTABLE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390670
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #387 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/387/ ) svn merge -c 1379022 FIXES: MAPREDUCE-4610 . Support deprecated mapreduce.job.counters.limit property in MR2. (Revision 1390670) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390670 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1182 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1182/)
        MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in MR2. (Revision 1379022)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379022
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1182 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1182/ ) MAPREDUCE-4610 . Support deprecated mapreduce.job.counters.limit property in MR2. (Revision 1379022) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379022 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1151 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1151/)
        MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in MR2. (Revision 1379022)

        Result = FAILURE
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379022
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1151 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1151/ ) MAPREDUCE-4610 . Support deprecated mapreduce.job.counters.limit property in MR2. (Revision 1379022) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379022 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.2.0-alpha [ 12322471 ]
        Resolution Fixed [ 1 ]
        Hide
        Tom White added a comment -

        I just committed this.

        Show
        Tom White added a comment - I just committed this.
        Hide
        Philip Zeyliger added a comment -

        +1 LGTM.

        Show
        Philip Zeyliger added a comment - +1 LGTM.
        Hide
        Alejandro Abdelnur added a comment -

        +1. LGTM.

        Show
        Alejandro Abdelnur added a comment - +1. LGTM.
        Hide
        Tom White added a comment -

        The extra javac warning is from the Configuration.addDeprecation which follows the style of the other ones in ConfigUtil, so is harmless.

        Show
        Tom White added a comment - The extra javac warning is from the Configuration.addDeprecation which follows the style of the other ones in ConfigUtil, so is harmless.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12543096/MAPREDUCE-4610.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The applied patch generated 2059 javac compiler warnings (more than the trunk's current 2058 warnings).

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2791//testReport/
        Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2791//artifact/trunk/patchprocess/diffJavacWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2791//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543096/MAPREDUCE-4610.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac. The applied patch generated 2059 javac compiler warnings (more than the trunk's current 2058 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2791//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2791//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2791//console This message is automatically generated.
        Tom White made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tom White made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-4610.patch [ 12543096 ]
        Hide
        Tom White added a comment -

        Here's a patch that fixes this. I tested it manually by starting a cluster and running a job that creates more than the default number of counters (120). When I increased mapreduce.job.counters.limit beyond that number I didn't see any counter limit exception.

        Writing a unit test is actually difficult since the limits are loaded statically. I'd like to fix that as a part of fixing the client-side limit enforcement problem in MAPREDUCE-3936. But I think this is useful to go in as a compatibility fix first.

        Show
        Tom White added a comment - Here's a patch that fixes this. I tested it manually by starting a cluster and running a job that creates more than the default number of counters (120). When I increased mapreduce.job.counters.limit beyond that number I didn't see any counter limit exception. Writing a unit test is actually difficult since the limits are loaded statically. I'd like to fix that as a part of fixing the client-side limit enforcement problem in MAPREDUCE-3936 . But I think this is useful to go in as a compatibility fix first.
        Tom White created issue -

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development