Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1100 User's task-logs filling up local disks on the TaskTrackers
  3. MAPREDUCE-927

Cleanup of task-logs should happen in TaskTracker instead of the Child

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.22.0
    • security, tasktracker
    • None
    • Reviewed
    • Hide
      Moved Task log cleanup into a separate thread in TaskTracker.
      Added configuration "mapreduce.job.userlog.retain.hours" to specify the time(in hours) for which the user-logs are to be retained after the job completion.
      Show
      Moved Task log cleanup into a separate thread in TaskTracker. Added configuration "mapreduce.job.userlog.retain.hours" to specify the time(in hours) for which the user-logs are to be retained after the job completion.

    Description

      Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons: 1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will get affected due to operations that TT actually should own.

      Attachments

        1. patch-927-5-dist.txt
          152 kB
          Vinod Kumar Vavilapalli
        2. patch-927-4.txt
          50 kB
          Amareshwari Sriramadasu
        3. patch-927-3.txt
          51 kB
          Amareshwari Sriramadasu
        4. patch-927-2.txt
          43 kB
          Amareshwari Sriramadasu
        5. patch-927-1.txt
          31 kB
          Amareshwari Sriramadasu
        6. patch-927.txt
          31 kB
          Amareshwari Sriramadasu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            amareshwari Amareshwari Sriramadasu
            vinodkv Vinod Kumar Vavilapalli
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment