Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5499

JobTracker metrics do not match job counters

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.20.0
    • Component/s: metrics
    • Labels:
      None

      Description

      After one run of a 10 map, 3 reduce sort job, the mapred metrics report:

      mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
        reduceTaskSlots=3, reduces_running=0, tasks_completed=15, tasks_failed_ping=0, tasks_failed_timeout=0
      mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=1, jobs_submitted=1, \
        maps_completed=10, maps_launched=12, reduces_completed=3, reduces_launched=3
      

      After the second (w/ one manually killed reduce)

      mapred.tasktracker: hostName=snip, sessionId=, mapTaskSlots=3, maps_running=0, \
         reduceTaskSlots=3, reduces_running=0, tasks_completed=32, tasks_failed_ping=0, tasks_failed_timeout=0
      mapred.jobtracker: hostName=snip, sessionId=, jobs_completed=2, jobs_submitted=2, \
        maps_completed=20, maps_launched=24, reduces_completed=6, reduces_launched=8
      

      The counters report the expected 10/3 map/reduce completions, the second job reporting a single failure.

      1. The maps_launched and reduces_launched counts are likely recording setup and cleanup tasks as well
      2. After being recorded among launched tasks, it looks like setup and cleanup are not included among completed tasks
      3. tasks_completed on the TaskTracker should include only user tasks
      4. The reduces_launched count makes little sense to me. With three reduces launched per job and one failed, what's the other launched reduce counting?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cdouglas Christopher Douglas
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: