[MAPREDUCE-5588] TaskTrackers get killed by JettyBugMonitor because of incredibly high cpu usage

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.1.2
• Component/s: tasktracker
• Labels: cpu-usage jetty tasktracker
• Assignee: Chris Nauroth
• Reporter: Anthony MOI
• Created: 17/Oct/13 13:17
• Updated: 17/Oct/13 17:10
• Resolved: 17/Oct/13 17:09

Description

We are running a little cluster with 10 servers running task trackers. All of them are getting killed randomly with the following message

2013-10-17 11:32:31,037 FATAL org.apache.hadoop.mapred.JettyBugMonitor: ************************************************************
Jetty CPU usage: 120093277.1%. This is greater than the fatal threshold mapred.tasktracker.jetty.cpu.threshold.fatal. Aborting JVM.
************************************************************
2013-10-17 11:32:31,039 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker
************************************************************/

Everytime, the message notices a cpu usage above 120M%. Everything has been running for a while now (since 1.1.2 release) without any problems, and it started just like that.

Any idea of what could cause this ?

Issue Links

• duplicates: MAPREDUCE-2980 Fetch failures and other related issues in Jetty 6.1.26
• relates to: MAPREDUCE-3184 Improve handling of fetch failures when a tasktracker is not responding on HTTP