Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1316

JobTracker holds stale references to retired jobs via unreported tasks

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      JobTracker holds stale references to TaskInProgress objects and hence indirectly holds reference to retired jobs resulting into memory leak. Only task-attempts which are yet to report their status are left behind in the memory. All the task-attempts are now removed from the JobTracker by iterating over the scheduled task-attempt ids instead of iterating over available task statuses.
      Show
      JobTracker holds stale references to TaskInProgress objects and hence indirectly holds reference to retired jobs resulting into memory leak. Only task-attempts which are yet to report their status are left behind in the memory. All the task-attempts are now removed from the JobTracker by iterating over the scheduled task-attempt ids instead of iterating over available task statuses.

      Description

      JobTracker fails to remove unreported tasks' mapping from taskToTIPMap if the job finishes and retires. Unreported tasks refers to tasks that were scheduled but the tasktracker did not report back with the task status. In such cases a stale reference is held to TaskInProgress (and thus JobInProgress) long after the job is gone leading to memory leak.

        Attachments

        1. mapreduce-1316-y20s.patch
          38 kB
          Hemanth Yamijala
        2. mapreduce-1316-v1.15-branch-0.21.patch
          32 kB
          Amar Kamat
        3. mapreduce-1316-v1.15.patch
          32 kB
          Amar Kamat
        4. mapreduce-1316-v1.15-branch20-yahoo.patch
          38 kB
          Arun C Murthy
        5. mapreduce-1316-v1.14.1-branch20-yahoo.patch
          38 kB
          Amar Kamat
        6. mapreduce-1316-v1.14-branch20-yahoo.patch
          37 kB
          Amar Kamat
        7. mapreduce-1316-v1.13-branch20-yahoo.patch
          21 kB
          Amar Kamat
        8. mapreduce-1316-v1.11.patch
          14 kB
          Amar Kamat
        9. mapreduce-1316-v1.7.patch
          15 kB
          Amar Kamat

          Issue Links

            Activity

              People

              • Assignee:
                amar_kamat Amar Kamat
                Reporter:
                amar_kamat Amar Kamat
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: