Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2037

Capturing interim progress times, CPU usage, and memory usage, when tasks reach certain progress thresholds

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      Capture intermediate task resource consumption information:
      * Time taken so far
      * CPU load [either at the time the data are taken, or exponentially smoothed]
      * Memory load [also either at the time the data are taken, or exponentially smoothed]

      This would be taken at intervals that depend on the task progress plateaus. For example, reducers have three progress ranges - [0-1/3], (1/3-2/3], and (2/3-3/3] - where fundamentally different activities happen. Mappers have different boundaries that are not symmetrically placed [0-9/10], (9/10-1]. Data capture boundaries should coincide with activity boundaries. For the state information capture [CPU and memory] we should average over the covered interval.
      Show
      Capture intermediate task resource consumption information: * Time taken so far * CPU load [either at the time the data are taken, or exponentially smoothed] * Memory load [also either at the time the data are taken, or exponentially smoothed] This would be taken at intervals that depend on the task progress plateaus. For example, reducers have three progress ranges - [0-1/3], (1/3-2/3], and (2/3-3/3] - where fundamentally different activities happen. Mappers have different boundaries that are not symmetrically placed [0-9/10], (9/10-1]. Data capture boundaries should coincide with activity boundaries. For the state information capture [CPU and memory] we should average over the covered interval.

      Description

      We would like to capture the following information at certain progress thresholds as a task runs:

      • Time taken so far
      • CPU load [either at the time the data are taken, or exponentially smoothed]
      • Memory load [also either at the time the data are taken, or exponentially smoothed]

      This would be taken at intervals that depend on the task progress plateaus. For example, reducers have three progress ranges – [0-1/3], (1/3-2/3], and (2/3-3/3] – where fundamentally different activities happen. Mappers have different boundaries, I understand, that are not symmetrically placed. Data capture boundaries should coincide with activity boundaries. For the state information capture [CPU and memory] we should average over the covered interval.

      This data would flow in with the heartbeats. It would be placed in the job history as part of the task attempt completion event, so it could be processed by rumen or some similar tool and could drive a benchmark engine.

      1. MAPREDUCE-2037.patch
        82 kB
        Arun C Murthy
      2. MAPREDUCE-2037.patch
        90 kB
        Arun C Murthy

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          I didn't realize before but MAPREDUCE-220 captures the cpu/memory load at the time of task completion. So the core functionality is already there in trunk.

          But the load at the time of task completion isn't really a useful stat. +1 for either exponential smoothing or a simpler capturing of highest,lowest and average loads for cpu and memory.

          Show
          Vinod Kumar Vavilapalli added a comment - I didn't realize before but MAPREDUCE-220 captures the cpu/memory load at the time of task completion. So the core functionality is already there in trunk. But the load at the time of task completion isn't really a useful stat. +1 for either exponential smoothing or a simpler capturing of highest,lowest and average loads for cpu and memory.
          Hide
          Hong Tang added a comment -

          -1 on using EWMA to capture CPU usage. It is more useful to track the aggregated cpu tick counter as raw data, and we can always calculate EWMA from that later, but not vice versa. It'd be also useful to capture the number of threads that are included in the calculation. So each entry looks like the following: <time, cpu-ticker-counter, #threads>. I'd also like to capture CPU MHz number for the task tracker so that I can know if we are saturating the CPU.

          Show
          Hong Tang added a comment - -1 on using EWMA to capture CPU usage. It is more useful to track the aggregated cpu tick counter as raw data, and we can always calculate EWMA from that later, but not vice versa. It'd be also useful to capture the number of threads that are included in the calculation. So each entry looks like the following: <time, cpu-ticker-counter, #threads>. I'd also like to capture CPU MHz number for the task tracker so that I can know if we are saturating the CPU.
          Hide
          Dick King added a comment -

          Benchmarks to support more realistic validation of putative scheduler improvements would benefit from a gridmix3-like tool that can simulate the CPU usage patterns of the tasks of the emulated jobs. That includes both the average loads of the various tasks, and also the time variation. In order to develop this information, we need to capture the CPU usage of each task over time.

          Fortunately, on linux systems, there's a way to capture this. The /proc/n/stat information appears to capture everything I need.

          I would plumb this using LinuxResourceCalculatorPlugin and TaskStatus .

          The information will be placed in the job history files, in the task attempt end records. This might be placed as a coded character string with a few dozen characters.

          Show
          Dick King added a comment - Benchmarks to support more realistic validation of putative scheduler improvements would benefit from a gridmix3-like tool that can simulate the CPU usage patterns of the tasks of the emulated jobs. That includes both the average loads of the various tasks, and also the time variation. In order to develop this information, we need to capture the CPU usage of each task over time. Fortunately, on linux systems, there's a way to capture this. The /proc/n/stat information appears to capture everything I need. I would plumb this using LinuxResourceCalculatorPlugin and TaskStatus . The information will be placed in the job history files, in the task attempt end records. This might be placed as a coded character string with a few dozen characters.
          Hide
          Dick King added a comment -

          Worker tasks seldom have multiple threads. Streaming and its friends spawn a task, and of course users can write whatever code they want, but most tasks burn their CPU time in their sole thread.

          Of course, when we do have streaming we need to capture the info from the slave task...

          Show
          Dick King added a comment - Worker tasks seldom have multiple threads. Streaming and its friends spawn a task, and of course users can write whatever code they want, but most tasks burn their CPU time in their sole thread. Of course, when we do have streaming we need to capture the info from the slave task...
          Hide
          Dick King added a comment -

          The information I need flows with the counters as a result of MAPREDUCE-220 .

          Counters ship from the TaskTracker to the JobTracker in the heartbeat – but prior to 901 this doesn't happen every heartbeat. Since the purpose of this code is to approximate a graph of resource use v. progress from snapshots that come with the heartbeats, we'll have to interpolate more aggressively and will lose accuracy when some counter instances are not shipped.

          With 901, the framework data will be shipped every time.

          Show
          Dick King added a comment - The information I need flows with the counters as a result of MAPREDUCE-220 . Counters ship from the TaskTracker to the JobTracker in the heartbeat – but prior to 901 this doesn't happen every heartbeat. Since the purpose of this code is to approximate a graph of resource use v. progress from snapshots that come with the heartbeats, we'll have to interpolate more aggressively and will lose accuracy when some counter instances are not shipped. With 901, the framework data will be shipped every time.
          Hide
          Dick King added a comment -

          The information we need is already being gathered. Moving it where it needs to go is a small matter of plumbing.

          The info flows into the job tracker via counters. I'll need to check whether accessing the counters will be too expensive to do within heartbeat [inside the {{JobTracker}} lock]. I might end up creating a queue of unprocessed counter blocks that can be handled in another thread.

          Show
          Dick King added a comment - The information we need is already being gathered. Moving it where it needs to go is a small matter of plumbing. The info flows into the job tracker via counters. I'll need to check whether accessing the counters will be too expensive to do within heartbeat [inside the {{JobTracker}} lock] . I might end up creating a queue of unprocessed counter blocks that can be handled in another thread.
          Hide
          Arun C Murthy added a comment -

          Patch ported from y-merge branch for ensuring we can merge MAPREDUCE-279 to trunk. Credit, of course, goes to Dick.

          Show
          Arun C Murthy added a comment - Patch ported from y-merge branch for ensuring we can merge MAPREDUCE-279 to trunk. Credit, of course, goes to Dick.
          Hide
          Arun C Murthy added a comment -

          Minor update, messed up diff previously.

          Show
          Arun C Murthy added a comment - Minor update, messed up diff previously.
          Hide
          Arun C Murthy added a comment -

          I just committed this after manually running tests. Thanks Dick!

          Show
          Arun C Murthy added a comment - I just committed this after manually running tests. Thanks Dick!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #764 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/764/)
          MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for tasks. Contributed by Dick King.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157253
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/AvroArrayUtils.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapTaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/java/mapred-default.xml
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/Events.avpr
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/StatePeriodicStats.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskPerformanceSplits.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/CumulativePeriodicStats.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceTaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/PeriodicStatsAccumulator.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/ProgressSplitsBlock.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #764 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/764/ ) MAPREDUCE-2037 . Capture intermediate progress, CPU and memory usage for tasks. Contributed by Dick King. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157253 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/AvroArrayUtils.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapTaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/java/mapred-default.xml /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/Events.avpr /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/StatePeriodicStats.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskPerformanceSplits.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/CumulativePeriodicStats.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceTaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/PeriodicStatsAccumulator.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/ProgressSplitsBlock.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/754/)
          MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for tasks. Contributed by Dick King.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157253
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/AvroArrayUtils.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapTaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/java/mapred-default.xml
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/Events.avpr
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/StatePeriodicStats.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskPerformanceSplits.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/CumulativePeriodicStats.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceTaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/PeriodicStatsAccumulator.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/ProgressSplitsBlock.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/754/ ) MAPREDUCE-2037 . Capture intermediate progress, CPU and memory usage for tasks. Contributed by Dick King. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157253 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/AvroArrayUtils.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapTaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/java/mapred-default.xml /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/Events.avpr /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/StatePeriodicStats.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskPerformanceSplits.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/CumulativePeriodicStats.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceTaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/PeriodicStatsAccumulator.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/ProgressSplitsBlock.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #742 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/742/)
          MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for tasks. Contributed by Dick King.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157253
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/AvroArrayUtils.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapTaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/java/mapred-default.xml
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/Events.avpr
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/StatePeriodicStats.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskPerformanceSplits.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/CumulativePeriodicStats.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceTaskAttemptInfo.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/PeriodicStatsAccumulator.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/ProgressSplitsBlock.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java
          • /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #742 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/742/ ) MAPREDUCE-2037 . Capture intermediate progress, CPU and memory usage for tasks. Contributed by Dick King. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157253 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/AvroArrayUtils.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapTaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/java/mapred-default.xml /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/TaskInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/Counters.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/Events.avpr /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/StatePeriodicStats.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/ReduceAttemptFinishedEvent.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestTaskPerformanceSplits.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ZombieJob.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceAttempt20LineHistoryEventEmitter.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEvents.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/CumulativePeriodicStats.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/ReduceTaskAttemptInfo.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/TaskAttempt20LineEventEmitter.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/JobInProgress.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/PeriodicStatsAccumulator.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapred/ProgressSplitsBlock.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/MapAttemptFinishedEvent.java /hadoop/common/trunk/mapreduce/src/tools/org/apache/hadoop/tools/rumen/MapAttempt20LineHistoryEventEmitter.java
          Hide
          Arun C Murthy added a comment -

          Editorial pass over hadoop-0.23 content.

          Show
          Arun C Murthy added a comment - Editorial pass over hadoop-0.23 content.

            People

            • Assignee:
              Dick King
              Reporter:
              Dick King
            • Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development