Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-964

Inaccurate values in jobSummary logs

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • None
    • None
    • Reviewed
    • Updates to task timing information were fixed in some code paths that led to inconsistent metering of task run times. Also, checks were added to guard against inconsistencies and record information about state if they should happen to occur.

    Description

      For some jobs the mapSlotSeconds is incorrect.

      negative value

      09/09/01 18:31:44 INFOmapred.JobInProgress$JobSummary: jobId=job_200908270718_4568,submitTime=1251823543976,launchTime=1251823554310,finishTime=1251829904565, numMaps=7965,numSlotsPerMap=1,numReduces=40,numSlotsPerReduce=1,user=wile,queue=runner,status=SUCCEEDED, mapSlotSeconds=-2503133523,reduceSlotsSeconds=186536,clusterMapCapacity=11262,clusterReduceCapacity=3754

      or too high

      09/09/02 23:59:57 INFO mapred.JobInProgress$JobSummary: jobId=job_200908270718_5861,submitTime=1251935672924,launchTime=1251935687698,finishTime=1251935997949, numMaps=1026,numSlotsPerMap=1,numReduces=10,numSlotsPerReduce=1,user=dfsload,queue=gridops,status=SUCCEEDED,
      mapSlotSeconds=1251949742,reduceSlotsSeconds=537,clusterMapCapacity=11262,clusterReduceCapacity=3754

      Attachments

        1. mapreduce-964-ydist-1.patch
          0.5 kB
          Sreekanth Ramakrishnan
        2. MAPREDUCE-964-4.patch
          7 kB
          Hemanth Yamijala
        3. mapreduce-964-3.patch
          6 kB
          Sreekanth Ramakrishnan
        4. mapreduce-964-ydist.patch
          6 kB
          Sreekanth Ramakrishnan
        5. mapreduce-964-2.patch
          6 kB
          Sreekanth Ramakrishnan
        6. mapreduce-964-ydist.patch
          5 kB
          Sreekanth Ramakrishnan
        7. mapreduce-964-1.patch
          2 kB
          Sreekanth Ramakrishnan

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sreekanth Sreekanth Ramakrishnan Assign to me
            rajive Rajiv Chittajallu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment