Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3749

ConcurrentModificationException in counter groups

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException.

      This was found while running Hive unit tests against a recent 0.23 version.

      1. MAPREDUCE-3749-test.patch
        1 kB
        Tom White
      2. MAPREDUCE-3749-concurrent.patch
        5 kB
        Tom White
      3. MAPREDUCE-3749.patch
        5 kB
        Tom White
      4. MAPREDUCE-3749.patch
        5 kB
        Tom White

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          Here's a test that exposes the problem.

          Show
          Tom White added a comment - Here's a test that exposes the problem.
          Hide
          Tom White added a comment -

          Here's a fix which uses Guava's ImmutableSet.copyOf before creating the iterator. There is a similar problem in AbstractCounterGroup too, which I've fixed in the same way.

          However, it may be better to fix these problems by using a concurrent collection rather than synchronization and copying. MAPREDUCE-3511 was an example of having too many copies of counter data structures in memory.

          Getting weakly consistent views of counters is consistent with their current semantics during the course of a job. So we might use ConcurrentSkipListMap for the maps of name -> counter/group. Alternatively, we could use ConcurrentHashMap and do sorting by name in the view layer.

          Show
          Tom White added a comment - Here's a fix which uses Guava's ImmutableSet.copyOf before creating the iterator. There is a similar problem in AbstractCounterGroup too, which I've fixed in the same way. However, it may be better to fix these problems by using a concurrent collection rather than synchronization and copying. MAPREDUCE-3511 was an example of having too many copies of counter data structures in memory. Getting weakly consistent views of counters is consistent with their current semantics during the course of a job. So we might use ConcurrentSkipListMap for the maps of name -> counter/group. Alternatively, we could use ConcurrentHashMap and do sorting by name in the view layer.
          Hide
          Tom White added a comment -

          Same patch but rooted in the correct directory this time.

          Show
          Tom White added a comment - Same patch but rooted in the correct directory this time.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512462/MAPREDUCE-3749.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 3715 javac compiler warnings (more than the trunk's current 3713 warnings).

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1711//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1711//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512462/MAPREDUCE-3749.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3715 javac compiler warnings (more than the trunk's current 3713 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1711//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1711//console This message is automatically generated.
          Hide
          Tom White added a comment -

          Here's a version that uses ConcurrentSkipListMap.

          Show
          Tom White added a comment - Here's a version that uses ConcurrentSkipListMap.
          Hide
          Arun C Murthy added a comment -

          +1 lgtm

          Show
          Arun C Murthy added a comment - +1 lgtm
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512490/MAPREDUCE-3749-concurrent.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 3715 javac compiler warnings (more than the trunk's current 3713 warnings).

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1714//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1714//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512490/MAPREDUCE-3749-concurrent.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3715 javac compiler warnings (more than the trunk's current 3713 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1714//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1714//console This message is automatically generated.
          Hide
          Tom White added a comment -

          Thanks for taking a look, Arun. Which patch does your +1 apply to?

          Show
          Tom White added a comment - Thanks for taking a look, Arun. Which patch does your +1 apply to?
          Hide
          Arun C Murthy added a comment -

          The one with the a copy, that should suffice for now?

          Show
          Arun C Murthy added a comment - The one with the a copy, that should suffice for now?
          Hide
          Tom White added a comment -

          I just committed this. We can revisit the more concurrent approach (e.g. with ConcurrentSkipListMap) in the future.

          Show
          Tom White added a comment - I just committed this. We can revisit the more concurrent approach (e.g. with ConcurrentSkipListMap) in the future.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1696 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1696/)
          MAPREDUCE-3749. ConcurrentModificationException in counter groups.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1696 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1696/ ) MAPREDUCE-3749 . ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #444 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/444/)
          Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #444 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/444/ ) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #454 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/454/)
          Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #454 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/454/ ) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1625 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1625/)
          MAPREDUCE-3749. ConcurrentModificationException in counter groups.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1625 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1625/ ) MAPREDUCE-3749 . ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #468 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/468/)
          Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #468 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/468/ ) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1642 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1642/)
          MAPREDUCE-3749. ConcurrentModificationException in counter groups.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1642 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1642/ ) MAPREDUCE-3749 . ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #943 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/)
          MAPREDUCE-3749. ConcurrentModificationException in counter groups.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #943 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/ ) MAPREDUCE-3749 . ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #156 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/)
          Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #156 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/ ) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #178 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/)
          Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #178 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/ ) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238735 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #976 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/)
          MAPREDUCE-3749. ConcurrentModificationException in counter groups.

          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #976 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/ ) MAPREDUCE-3749 . ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238734 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Tom, FYI, we found that the extra copy is indeed a problem as we ran into performance issues while running benchmarks. I created MAPREDUCE-3827 to instead use a Concurrent HashMap and give a fail-safe weakly consistent iterator.

          Show
          Vinod Kumar Vavilapalli added a comment - Tom, FYI, we found that the extra copy is indeed a problem as we ran into performance issues while running benchmarks. I created MAPREDUCE-3827 to instead use a Concurrent HashMap and give a fail-safe weakly consistent iterator.

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development