[MAPREDUCE-1100] User's task-logs filling up local disks on the TaskTrackers

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.20.1, 0.20.2, 0.21.0
• Fix Version/s: 0.21.1, 0.22.1
• Component/s: tasktracker
• Labels: critical-0.22.0
• Target Version/s: 0.22.0
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Vinod Kumar Vavilapalli
• Created: 13/Oct/09 09:13
• Updated: 12/Dec/11 06:09

Description

Some user's jobs are filling up TT disks by outrageous logging. mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are getting filled up before task-log cleanup via mapred.task.userlog.retain.hours can kick in.

Sub-Tasks

• Cleanup of task-logs should happen in TaskTracker instead of the Child: Closed
• Truncate logs of finished tasks to prevent node thrash due to excessive logging: Open