Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5351

JobTracker memory leak caused by CleanupQueue reopening FileSystem

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 1.1.2
    • 1-win, 1.2.1
    • jobtracker
    • None

    Description

      When a job is completed, closeAllForUGI is called to close all the cached FileSystems in the FileSystem cache. However, the CleanupQueue may run after this occurs and call FileSystem.get() to delete the staging directory, adding a FileSystem to the cache that will never be closed.

      People on the user-list have reported this causing their JobTrackers to OOME every two weeks.

      Attachments

        1. MAPREDUCE-5351-addendum-1.patch
          2 kB
          Sandy Ryza
        2. MAPREDUCE-5351-addendum.patch
          0.6 kB
          Sandy Ryza
        3. MAPREDUCE-5351-2.patch
          6 kB
          Sandy Ryza
        4. MAPREDUCE-5351-1.patch
          6 kB
          Sandy Ryza
        5. MAPREDUCE-5351.patch
          3 kB
          Sandy Ryza
        6. JobInProgress_JobHistory.patch
          3 kB
          jandyfish

        Issue Links

          Activity

            sandyr Sandy Ryza created issue -
            sandyr Sandy Ryza made changes -
            Field Original Value New Value
            Attachment MAPREDUCE-5351.patch [ 12589526 ]
            sandyr Sandy Ryza made changes -
            Attachment MAPREDUCE-5351-1.patch [ 12590103 ]
            sandyr Sandy Ryza made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            sandyr Sandy Ryza made changes -
            Attachment MAPREDUCE-5351-2.patch [ 12590105 ]
            acmurthy Arun Murthy made changes -
            Fix Version/s 1.2.1 [ 12324149 ]
            Resolution Fixed [ 1 ]
            Status Patch Available [ 10002 ] Resolved [ 5 ]
            arpitgupta Arpit Gupta made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            sandyr Sandy Ryza made changes -
            Attachment MAPREDUCE-5351-addendum.patch [ 12590494 ]
            sandyr Sandy Ryza made changes -
            Status Reopened [ 4 ] Patch Available [ 10002 ]
            sandyr Sandy Ryza made changes -
            Attachment MAPREDUCE-5351-addendum-1.patch [ 12590672 ]
            mattf Matthew Foley made changes -
            Resolution Fixed [ 1 ]
            Status Patch Available [ 10002 ] Resolved [ 5 ]
            mattf Matthew Foley made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            xifang Douma Fang made changes -
            Link This issue relates to MAPREDUCE-5508 [ MAPREDUCE-5508 ]
            cnauroth Chris Nauroth made changes -
            Fix Version/s 1-win [ 12321744 ]
            jandyfish jandyfish made changes -
            Attachment JobInProgress_JobHistory.patch [ 12613092 ]
            yunsh Sanghyun Yun made changes -
            Assignee Sandy Ryza [ sandyr ] Sanghyun Yun [ yunsh ]
            yunsh Sanghyun Yun made changes -
            Assignee Sanghyun Yun [ yunsh ] Sandy Ryza [ sandyr ]

            People

              sandyr Sandy Ryza
              sandyr Sandy Ryza
              Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: