Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1100 User's task-logs filling up local disks on the TaskTrackers
  3. MAPREDUCE-927

Cleanup of task-logs should happen in TaskTracker instead of the Child

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.22.0
    • security, tasktracker
    • None
    • Reviewed
    • Hide
      Moved Task log cleanup into a separate thread in TaskTracker.
      Added configuration "mapreduce.job.userlog.retain.hours" to specify the time(in hours) for which the user-logs are to be retained after the job completion.
      Show
      Moved Task log cleanup into a separate thread in TaskTracker. Added configuration "mapreduce.job.userlog.retain.hours" to specify the time(in hours) for which the user-logs are to be retained after the job completion.

    Description

      Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons: 1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will get affected due to operations that TT actually should own.

      Attachments

        1. patch-927.txt
          31 kB
          Amareshwari Sriramadasu
        2. patch-927-1.txt
          31 kB
          Amareshwari Sriramadasu
        3. patch-927-2.txt
          43 kB
          Amareshwari Sriramadasu
        4. patch-927-3.txt
          51 kB
          Amareshwari Sriramadasu
        5. patch-927-4.txt
          50 kB
          Amareshwari Sriramadasu
        6. patch-927-5-dist.txt
          152 kB
          Vinod Kumar Vavilapalli

        Issue Links

          Activity

            People

              amareshwari Amareshwari Sriramadasu
              vinodkv Vinod Kumar Vavilapalli
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: