Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-901

Move Framework Counters into a TaskMetric structure

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: task
    • Labels:
      None
    • Release Note:
      Efficient implementation of MapReduce framework counters.

      Description

      I think we should move all of the Counters that the framework updates into a single class called TaskMetrics. TaskMetrics would have specific fields for each of the metrics like input records, input bytes, output records, etc.

      It would both reduce the serialized size of the heartbeats (by shrinking the Counters down to just the user's counters) and decrease the latency for updates to the JobTracker (since Counters are sent at most 1/minute instead of 1/heartbeat).

      1. mr-901-trunk-v1.patch
        104 kB
        Luke Lu
      2. MAPREDUCE-901.patch_2
        292 kB
        Arun C Murthy
      3. MAPREDUCE-901.patch
        24 kB
        Arun C Murthy
      4. MAPREDUCE-901.patch
        60 kB
        Arun C Murthy
      5. MAPREDUCE-901.patch
        156 kB
        Arun C Murthy
      6. FrameworkCounterGroup.java
        4 kB
        Luke Lu
      7. 901_1.patch
        49 kB
        Devaraj Das
      8. 901_1.patch
        54 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #982 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/)
          MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #982 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/ ) MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #184 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/)
          Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #184 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/ ) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #162 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/)
          Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #162 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/ ) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #949 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/)
          MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #949 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/ ) MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #514 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/514/)
          Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #514 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/514/ ) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1692 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1692/)
          MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1692 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1692/ ) MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #506 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/506/)
          Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #506 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/506/ ) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1679 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1679/)
          MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1679 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1679/ ) MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1752 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1752/)
          MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1752 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1752/ ) MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241319 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #496 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/496/)
          Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #496 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/496/ ) Merge -c 1241319 from trunk to branch-0.23 to fix MAPREDUCE-3697 . Support binary compatibility for Counters after MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241321 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #742 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/742/)
          Fixed bad commit for MAPREDUCE-901.
          MAPREDUCE-901. Efficient framework counters. Contributed by Luke Lu.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157454
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157290
          Files :

          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskStatus.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/InterTrackerProtocol.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/LimitExceededException.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/TestCounters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/JobCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/Limits.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/package-info.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/CounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/CountersStrings.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupFactory.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRDFSSort.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #742 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/742/ ) Fixed bad commit for MAPREDUCE-901 . MAPREDUCE-901 . Efficient framework counters. Contributed by Luke Lu. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157454 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157290 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskStatus.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/InterTrackerProtocol.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/LimitExceededException.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/TestCounters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/JobCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/Limits.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/package-info.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/CounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/CountersStrings.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupFactory.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRDFSSort.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/754/)
          Fixed bad commit for MAPREDUCE-901.
          MAPREDUCE-901. Efficient framework counters. Contributed by Luke Lu.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157454
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157290
          Files :

          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskStatus.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/InterTrackerProtocol.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/LimitExceededException.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/TestCounters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/JobCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/Limits.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/package-info.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/CounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/CountersStrings.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupFactory.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRDFSSort.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/754/ ) Fixed bad commit for MAPREDUCE-901 . MAPREDUCE-901 . Efficient framework counters. Contributed by Luke Lu. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157454 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157290 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskStatus.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/InterTrackerProtocol.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/LimitExceededException.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/TestCounters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/JobCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/Limits.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/package-info.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/CounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/CountersStrings.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupFactory.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRDFSSort.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #767 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/767/)
          Fixed bad commit for MAPREDUCE-901.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157454
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #767 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/767/ ) Fixed bad commit for MAPREDUCE-901 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157454 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig
          Hide
          Arun C Murthy added a comment -

          Thanks for pointing it out Alexey - my bad, fixed now.

          Show
          Arun C Murthy added a comment - Thanks for pointing it out Alexey - my bad, fixed now.
          Hide
          Alexey Diomin added a comment -

          1. after commit new file created /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig
          2. bad code for logging (JobInProgress.java:1253)

           
            public synchronized Counters getJobCounters() {
              try {
                throw new IOException("");
              } catch (IOException ioe) {
                LOG.info("getJC", ioe);
              }
              return jobCounters;
            }
          
          Show
          Alexey Diomin added a comment - 1. after commit new file created /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig 2. bad code for logging (JobInProgress.java:1253) public synchronized Counters getJobCounters() { try { throw new IOException(""); } catch (IOException ioe) { LOG.info( "getJC" , ioe); } return jobCounters; }
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #765 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/765/)
          MAPREDUCE-901. Efficient framework counters. Contributed by Luke Lu.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157290
          Files :

          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskStatus.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/InterTrackerProtocol.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/LimitExceededException.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/TestCounters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/JobCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/Limits.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/package-info.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/CounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/CountersStrings.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupFactory.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRDFSSort.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #765 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/765/ ) MAPREDUCE-901 . Efficient framework counters. Contributed by Luke Lu. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157290 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskStatus.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/InterTrackerProtocol.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/LimitExceededException.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/TestCounters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/JobCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/Limits.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/package-info.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupBase.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java.orig /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/CounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/CountersStrings.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/GenericCounter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/CounterGroupFactory.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/FileSystemCounter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestSeveral.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMiniMRDFSSort.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/TaskCounter.properties /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/counters/AbstractCounter.java
          Hide
          Arun C Murthy added a comment -

          Thanks Tom.

          I just committed this. Thanks Luke!

          Show
          Arun C Murthy added a comment - Thanks Tom. I just committed this. Thanks Luke!
          Hide
          Tom White added a comment -

          +1 to addressing the points I raised in a follow-on JIRA.

          Show
          Tom White added a comment - +1 to addressing the points I raised in a follow-on JIRA.
          Hide
          Arun C Murthy added a comment -

          Updated patch to fix a couple of unit tests, which now pass.

          I think with MAPREDUCE-279, the counter limits are less interesting given that they do not affect other jobs - unlike MRv1 where this would affect the JobTracker. For now, I propose we commit this and re-visit Counter limits via a follow-on (blocker) jira. Thoughts?

          Show
          Arun C Murthy added a comment - Updated patch to fix a couple of unit tests, which now pass. I think with MAPREDUCE-279 , the counter limits are less interesting given that they do not affect other jobs - unlike MRv1 where this would affect the JobTracker. For now, I propose we commit this and re-visit Counter limits via a follow-on (blocker) jira. Thoughts?
          Hide
          Tom White added a comment -

          This is related to MAPREDUCE-2835, which makes the counter limits configurable in the 0.20-security branch.

          That branch already uses the mapreduce.job.counters.limit property to set the maximum number of counters allowed in a job. It's probably worth supporting it as a deprecated equivalent of mapreduce.job.counters.max introduced in this patch.

          Also, the new properties should be added to mapred-default.xml.

          Show
          Tom White added a comment - This is related to MAPREDUCE-2835 , which makes the counter limits configurable in the 0.20-security branch. That branch already uses the mapreduce.job.counters.limit property to set the maximum number of counters allowed in a job. It's probably worth supporting it as a deprecated equivalent of mapreduce.job.counters.max introduced in this patch. Also, the new properties should be added to mapred-default.xml.
          Hide
          Luke Lu added a comment -

          The port, which includes smarter counter limits (exclude framework counters for limits) lgtm. Defer +1 to Jenkins. There are some improvements can be made to improve binary compatibility (for most existing job jars) that we did/does not promise, deferring to a separate JIRA if binary compatibility becomes an issue.

          Show
          Luke Lu added a comment - The port, which includes smarter counter limits (exclude framework counters for limits) lgtm. Defer +1 to Jenkins. There are some improvements can be made to improve binary compatibility (for most existing job jars) that we did/does not promise, deferring to a separate JIRA if binary compatibility becomes an issue.
          Hide
          Arun C Murthy added a comment -

          Patch ported from y-merge branch for ensuring we can merge MAPREDUCE-901 to trunk. Credit, of course, goes to Luke.

          Show
          Arun C Murthy added a comment - Patch ported from y-merge branch for ensuring we can merge MAPREDUCE-901 to trunk. Credit, of course, goes to Luke.
          Hide
          Luke Lu added a comment -

          The latest patch already handles JobCounter and TaskCounter optimization (with the generic FrameworkCounterGroup) transparently. But it doesn't address file system counter optimization yet. However using concrete fs enums (hdfs, s3 etc.) like in the previous patches is too brittle, as the whole mapreduce package needs to be recompiled/released for every new implementation of distributed filesystem, which defeats the purpose of having a filesystem interface, where we can already query for (fs scheme, stats) tuples. HADOOP-4188 tried to address the issue but the treatment is incomplete: the Task#getFileSystemCounters helper method is package private and quite awkward to use: requires explict array indexing, e.g. getFileSystemCounters(scheme)[0] to return <SCHEME>_BYTES_READ (e.g. HDFS_BYTES_READ) to use with the generic counter interface. This also makes decoupled file system counter display name localization impossible.

          I propose that we add a file system counter API to the Counters framework. Something like:

          Counter getFileSystemCounter(String scheme, FileSystemCounter key);
          

          where FileSystemCounter is an enum class:

          public enum FileSystemCounter {
            BYTES_READ,
            BYTES_WRITTEN
            // etc.
          }
          

          We can take advantage of this interface to create an efficient file system counter group that can be more efficiently stored in memory and serialized (say: (<scheme>, vint(BYTES_READ), vint(BYTES_WRITTEN)...) tuples)

          Thoughts?

          Show
          Luke Lu added a comment - The latest patch already handles JobCounter and TaskCounter optimization (with the generic FrameworkCounterGroup) transparently. But it doesn't address file system counter optimization yet. However using concrete fs enums (hdfs, s3 etc.) like in the previous patches is too brittle, as the whole mapreduce package needs to be recompiled/released for every new implementation of distributed filesystem, which defeats the purpose of having a filesystem interface, where we can already query for (fs scheme, stats) tuples. HADOOP-4188 tried to address the issue but the treatment is incomplete: the Task#getFileSystemCounters helper method is package private and quite awkward to use: requires explict array indexing, e.g. getFileSystemCounters(scheme) [0] to return <SCHEME>_BYTES_READ (e.g. HDFS_BYTES_READ) to use with the generic counter interface. This also makes decoupled file system counter display name localization impossible. I propose that we add a file system counter API to the Counters framework. Something like: Counter getFileSystemCounter( String scheme, FileSystemCounter key); where FileSystemCounter is an enum class: public enum FileSystemCounter { BYTES_READ, BYTES_WRITTEN // etc. } We can take advantage of this interface to create an efficient file system counter group that can be more efficiently stored in memory and serialized (say: (<scheme>, vint(BYTES_READ), vint(BYTES_WRITTEN)...) tuples) Thoughts?
          Hide
          Luke Lu added a comment -

          v1 patch passes all existing counters tests, probably good enough to bother hudson.

          This is patch is basically a refactor of the counters framework. Blame java generics for the syntax

          Highlight of the patch:

          • No (intentional) user API changes.
          • Make mapred.Counters.Counter (legacy) and mapreduce.Counter (new) abstract
            • the existing constructors are already package private, needed to fix one test to use proper API instead of 'new'
          • Make mapred.Counters.Group (legacy) and mapreduce.CounterGroup interface
            • the existing constructors are already package private, needed to fix EventReader to use proper API instead of 'new'
          • Implement AbstractCounters and generic counter group in mapreduce.AbstractCounterGroup and framework counter group in mapreduce.FrameworkCounterGroup
            • Framework group counters have efficient in memory (fixed arrays of long and light weight counter facade) and serialized ((vint ordinal, vint value)* tuples) representations. Framework counters can be easily declared in CounterGroupFactory.
          • mapred.Counters and mapreduce.Counters contain the delta of AbstractCounters and use mixins to adapt to different interface
            • Much of the generics dancing is to support type safe Iterable in both legacy and new Counters interface.
          Show
          Luke Lu added a comment - v1 patch passes all existing counters tests, probably good enough to bother hudson. This is patch is basically a refactor of the counters framework. Blame java generics for the syntax Highlight of the patch: No (intentional) user API changes. Make mapred.Counters.Counter (legacy) and mapreduce.Counter (new) abstract the existing constructors are already package private, needed to fix one test to use proper API instead of 'new' Make mapred.Counters.Group (legacy) and mapreduce.CounterGroup interface the existing constructors are already package private, needed to fix EventReader to use proper API instead of 'new' Implement AbstractCounters and generic counter group in mapreduce.AbstractCounterGroup and framework counter group in mapreduce.FrameworkCounterGroup Framework group counters have efficient in memory (fixed arrays of long and light weight counter facade) and serialized ((vint ordinal, vint value)* tuples) representations. Framework counters can be easily declared in CounterGroupFactory. mapred.Counters and mapreduce.Counters contain the delta of AbstractCounters and use mixins to adapt to different interface Much of the generics dancing is to support type safe Iterable in both legacy and new Counters interface.
          Hide
          Arun C Murthy added a comment -

          I'm proposing a new approach to this issue: refactor the counter framework (while keeping the existing api) so that it's possible to have different implementation of counter groups, one of which is the FrameworkCounterGroup (see attached file.)

          This seems reasonable... I would be interested in a more sketched out version of the patch. Btw, can you ensure (unit tests) that end-user code for JobSubmissionProtocol.getCounters has no regression? Thanks.

          Show
          Arun C Murthy added a comment - I'm proposing a new approach to this issue: refactor the counter framework (while keeping the existing api) so that it's possible to have different implementation of counter groups, one of which is the FrameworkCounterGroup (see attached file.) This seems reasonable... I would be interested in a more sketched out version of the patch. Btw, can you ensure (unit tests) that end-user code for JobSubmissionProtocol.getCounters has no regression? Thanks.
          Hide
          Luke Lu added a comment -

          Looking through the patches, it seems that the existing approach requires sweeping changes to internal apis across pretty much all mapreduce components.

          I'm proposing a new approach to this issue: refactor the counter framework (while keeping the existing api) so that it's possible to have different implementation of counter groups, one of which is the FrameworkCounterGroup (see attached file.)

          This way, we can achieve the benefit of more efficient implementation for framework counters without changing the client code.

          The problem is a little complicated due to need to support of both old and new counters interface (both marked public and stable.) I'm working on a more complete patch to minimize the code duplication between new and old counter code. But I'm confident that the resulting code will be more concise and general (to support any future framework counters in different groups.) than the existing approach.

          Show
          Luke Lu added a comment - Looking through the patches, it seems that the existing approach requires sweeping changes to internal apis across pretty much all mapreduce components. I'm proposing a new approach to this issue: refactor the counter framework (while keeping the existing api) so that it's possible to have different implementation of counter groups, one of which is the FrameworkCounterGroup (see attached file.) This way, we can achieve the benefit of more efficient implementation for framework counters without changing the client code. The problem is a little complicated due to need to support of both old and new counters interface (both marked public and stable.) I'm working on a more complete patch to minimize the code duplication between new and old counter code. But I'm confident that the resulting code will be more concise and general (to support any future framework counters in different groups.) than the existing approach.
          Hide
          Hong Tang added a comment -

          Like metrics, we should also clearly document the framework counters.

          Show
          Hong Tang added a comment - Like metrics, we should also clearly document the framework counters.
          Hide
          Arun C Murthy added a comment -

          This is almost there, sans two important pieces:

          1. It looks like I need to change ReduceContext to take TaskMetrics in-lieu of Counters.Counter for it's inputCounter; a change to a public api... sigh!
          2. As Devaraj noted, I need to fix JobInProgress to store the incoming TaskMetrics values in it's Counters.
          Show
          Arun C Murthy added a comment - This is almost there, sans two important pieces: It looks like I need to change ReduceContext to take TaskMetrics in-lieu of Counters.Counter for it's inputCounter; a change to a public api... sigh! As Devaraj noted, I need to fix JobInProgress to store the incoming TaskMetrics values in it's Counters.
          Hide
          Devaraj Das added a comment -

          Looked at the latest patch. One thing that needs to be done here is that the JobInProgress counters needs to be updated with the TaskMetrics long values..

          Show
          Devaraj Das added a comment - Looked at the latest patch. One thing that needs to be done here is that the JobInProgress counters needs to be updated with the TaskMetrics long values..
          Hide
          Arun C Murthy added a comment -

          Preliminary patch while I'm blocked by MAPREDUCE-917.

          Show
          Arun C Murthy added a comment - Preliminary patch while I'm blocked by MAPREDUCE-917 .
          Hide
          Devaraj Das added a comment -

          1. I'd replace the array of longs with explicit fields.

          Owen, is this to avoid the array indexing? I kept it this way since the TaskCounter enum already existed, and it seemed simple to have a one-to-one mapping from the enum to the array. Also, in the future if we add another element to the TaskCounter enum, we don't have to explicitly modify TaskMetrics since it would automatically handle that..

          2. I'd drop the booleans for whether it has been used.

          I kept this so as to retain the existing semantics where an unused counter is not displayed to the user (for e.g., reduce-shuffle-bytes is not displayed as part of map tasks' counters). But I am okay to do without the booleans..

          3. The serialization should just be to write the fields in order using vints.

          If we remove the booleans, then it will be just writing the long array elements which is already written as vlongs... If we decide to switch to explicit fields for the longs then this part would change slightly..

          4. updateFrameworkCounters should just update the counters by using the enum values explicitly.

          Again, as i pointed out in (1) above, it helps to iterate over TaskCounter enum.

          5. updateFrameworkCounters should set, not increment the related counter.

          Makes sense

          6. TaskMetrics should be a field in the TaskStatus.

          It already is. A little bit of cleanup may be required w.r.t passing the object around in the TaskStatus methods..

          Show
          Devaraj Das added a comment - 1. I'd replace the array of longs with explicit fields. Owen, is this to avoid the array indexing? I kept it this way since the TaskCounter enum already existed, and it seemed simple to have a one-to-one mapping from the enum to the array. Also, in the future if we add another element to the TaskCounter enum, we don't have to explicitly modify TaskMetrics since it would automatically handle that.. 2. I'd drop the booleans for whether it has been used. I kept this so as to retain the existing semantics where an unused counter is not displayed to the user (for e.g., reduce-shuffle-bytes is not displayed as part of map tasks' counters). But I am okay to do without the booleans.. 3. The serialization should just be to write the fields in order using vints. If we remove the booleans, then it will be just writing the long array elements which is already written as vlongs... If we decide to switch to explicit fields for the longs then this part would change slightly.. 4. updateFrameworkCounters should just update the counters by using the enum values explicitly. Again, as i pointed out in (1) above, it helps to iterate over TaskCounter enum. 5. updateFrameworkCounters should set, not increment the related counter. Makes sense 6. TaskMetrics should be a field in the TaskStatus. It already is. A little bit of cleanup may be required w.r.t passing the object around in the TaskStatus methods..
          Hide
          Owen O'Malley added a comment -

          Comments:
          1. I'd replace the array of longs with explicit fields.
          2. I'd drop the booleans for whether it has been used.
          3. The serialization should just be to write the fields in order using vints.
          4. updateFrameworkCounters should just update the counters by using the enum values explicitly.
          5. updateFrameworkCounters should set, not increment the related counter.
          6. TaskMetrics should be a field in the TaskStatus.

          Show
          Owen O'Malley added a comment - Comments: 1. I'd replace the array of longs with explicit fields. 2. I'd drop the booleans for whether it has been used. 3. The serialization should just be to write the fields in order using vints. 4. updateFrameworkCounters should just update the counters by using the enum values explicitly. 5. updateFrameworkCounters should set, not increment the related counter. 6. TaskMetrics should be a field in the TaskStatus.
          Hide
          Devaraj Das added a comment -

          The patch proabably needs some cleanup w.r.t passing the TaskMetrics object to the TaskStatus methods. Some of the methods may not require to be passed the TaskMetrics object..

          Show
          Devaraj Das added a comment - The patch proabably needs some cleanup w.r.t passing the TaskMetrics object to the TaskStatus methods. Some of the methods may not require to be passed the TaskMetrics object..
          Hide
          Devaraj Das added a comment -

          That was my bad. sigh
          Attached is the correct patch. The TaskMetrics has a Counters field but that's mostly to take care of counters that are related to the FileSystemCounters which depends on the FileSystem in use, etc.

          Show
          Devaraj Das added a comment - That was my bad. sigh Attached is the correct patch. The TaskMetrics has a Counters field but that's mostly to take care of counters that are related to the FileSystemCounters which depends on the FileSystem in use, etc.
          Hide
          Arun C Murthy added a comment -

          Hmm... at the risk of sounding completely lame, I can't seem to find the definition of TaskMetrics or TaskCounters - did you forget to do included that in the patch?

          From the description it seems like TaskMetrics is related to Counters, maybe I should wait to see the patch - anyway I was hoping TaskMetrics would be a Writable and isn't related to Counters at all.

          Show
          Arun C Murthy added a comment - Hmm... at the risk of sounding completely lame, I can't seem to find the definition of TaskMetrics or TaskCounters - did you forget to do included that in the patch? From the description it seems like TaskMetrics is related to Counters, maybe I should wait to see the patch - anyway I was hoping TaskMetrics would be a Writable and isn't related to Counters at all.
          Hide
          Devaraj Das added a comment -

          Attaching a patch for review. I am still testing the patch. Also, a little bit of cleanup is required especially w.r.t to naming variables/fields in the classes. I will do that in a follow up patch.

          Some points on the approach:
          1) Defined a class TaskMetrics that has methods for updating the counters defined in o.a.h.mapreduce.TaskCounter.java. It also provides a utility method to update framework Counters that aren't defined in TaskCounter.java. Examples of such counters are the counters that the framework defines in the countergroup FileSystemCounters. For the TaskCounter counters, the RPC is optimized. For the framework counters like the FileSystemCounters, RPC uses the Counters serialization.
          2) The above is serialized out as part of TaskStatus object in the heartbeats.
          3) In TaskInProgress.java, the TIP's Counters is updated with the above counters obtained in the heartbeat.

          Would really appreciate a review on this one.

          And yes, this looks like a good thing to have for the jiras MAPREDUCE-220 and MAPREDUCE-718.

          Show
          Devaraj Das added a comment - Attaching a patch for review. I am still testing the patch. Also, a little bit of cleanup is required especially w.r.t to naming variables/fields in the classes. I will do that in a follow up patch. Some points on the approach: 1) Defined a class TaskMetrics that has methods for updating the counters defined in o.a.h.mapreduce.TaskCounter.java. It also provides a utility method to update framework Counters that aren't defined in TaskCounter.java. Examples of such counters are the counters that the framework defines in the countergroup FileSystemCounters. For the TaskCounter counters, the RPC is optimized. For the framework counters like the FileSystemCounters, RPC uses the Counters serialization. 2) The above is serialized out as part of TaskStatus object in the heartbeats. 3) In TaskInProgress.java, the TIP's Counters is updated with the above counters obtained in the heartbeat. Would really appreciate a review on this one. And yes, this looks like a good thing to have for the jiras MAPREDUCE-220 and MAPREDUCE-718 .
          Hide
          Arun C Murthy added a comment -

          +1

          I think this is the right direction, we should be using this for collecting all sorts of task-metrics (e.g. MAPREDUCE-220).

          In the not too distant future we should we exposing TaskMetrics to the schedulers to allow them to take decisions based on tasks' resource utilization and do away with the statically configured map/reduce slots.

          Show
          Arun C Murthy added a comment - +1 I think this is the right direction, we should be using this for collecting all sorts of task-metrics (e.g. MAPREDUCE-220 ). In the not too distant future we should we exposing TaskMetrics to the schedulers to allow them to take decisions based on tasks' resource utilization and do away with the statically configured map/reduce slots.
          Hide
          Nigel Daley added a comment -

          +1. Should be easier to test. Don't think we're doing any/much metrics testing right now.

          Show
          Nigel Daley added a comment - +1. Should be easier to test. Don't think we're doing any/much metrics testing right now.

            People

            • Assignee:
              Luke Lu
              Reporter:
              Owen O'Malley
            • Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development