[HADOOP-165] long response times from task trackers under load

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.2.0

Environment

190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory

• Assignee: Unassigned
• Reporter: Owen O'Malley
• Created: 25/Apr/06 23:34
• Updated: 08/Jul/09 16:51
• Resolved: 24/May/07 06:12

Description

we are seeing very slow response times from the task tracker. I put in some instrumentation to measure how long each call took for the RPC.Sever code to run the method (so it does not include serialization/deserialization time). The top of the list (in ms) looks like:

7581 progress node1192
7022 ping node1192
5393 ping node1162
4854 progress node1162
4749 progress node1194
4709 ping node1194
3813 ping node1100
3486 ping node1190
3266 progress node1190
3187 progress node1265
3078 ping node1203
2972 progress node1203
2947 progress node1240
2889 progress node1100
2875 ping node1116
2843 ping node1189
2772 ping node1183
2737 ping node1110
2727 progress node1183
2710 ping node1123
2563 ping node1304
2527 progress node1144
2479 ping node1137
2476 progress node1304
2430 ping node1240
2416 ping node1144
2377 progress node1176
2339 ping node1109
2321 progress node1114
2311 ping node1157
2185 ping node1265
2185 ping node1109
2172 ping node1114
2145 progress node1109
2127 ping node1176
2083 progress node1189
2076 ping node1229
2073 progress node1188
2072 progress node1123
2048 ping node1161
2003 progress node1110
1989 ping node1180
1963 ping node1114

Issue Links

• relates to: HADOOP-160 sleeping with locks held