[HADOOP-1009] Infinite loop in chooseTarget

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.2
• Assignee: Dhruba Borthakur
• Reporter: Dhruba Borthakur
• Created: 12/Feb/07 17:47
• Updated: 08/Jul/09 16:42
• Resolved: 13/Feb/07 22:44

Description

I am seeing the namenode hangs with 99% CPU usage. The stack trace invariably shows the following stack trace.

• java.util.Random.next(int) @bci=35, line=141 (Compiled frame; information may be imprecise)
• java.util.Random.nextInt(int) @bci=40, line=255 (Compiled frame)
• org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseRandom(int, org.apache.hadoop.dfs.DatanodeDescriptor[], java.util.List, long, int, java.util.List) @bci=7, line=2986 (Interpreted frame)
• org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseTarget(int, org.apache.hadoop.dfs.DatanodeDescriptor, org.apache.hadoop.dfs.DatanodeDescriptor[], java.util.List, long, int, java.util.List) @bci=211, line=2810 (Interpreted frame)
• org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseTarget(int, org.apache.hadoop.dfs.DatanodeDescriptor, java.util.List, java.util.List, long) @bci=141, line=2764 (Interpreted frame)
• org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseTarget(int, org.apache.hadoop.dfs.DatanodeDescriptor, java.util.List, long) @bci=25, line=2716 (Interpreted frame)
• org.apache.hadoop.dfs.FSNamesystem.startFile(org.apache.hadoop.io.UTF8, org.apache.hadoop.io.UTF8, org.apache.hadoop.io.UTF8, boolean, short, long) @bci=579, line=709 (Interpreted frame)
• org.apache.hadoop.dfs.NameNode.create(java.lang.String, java.lang.String, java.lang.String, boolean, short, long) @bci=97, line=270 (Interpreted frame)
• sun.reflect.GeneratedMethodAccessor5.invoke(java.lang.Object, java.lang.Object[]) @bci=238 (Interpreted frame)
• sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame)
• java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=111, line=585 (Interpreted frame)
• org.apache.hadoop.ipc.RPC$Server.call(org.apache.hadoop.io.Writable) @bci=64, line=337 (Compiled frame)
• org.apache.hadoop.ipc.Server$Handler.run() @bci=351, line=538 (Interpreted frame)