[HADOOP-3537] Datanode not starting up with java.lang.StringIndexOutOfBoundsException in NetworkTopology.remove

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.17.0
• Fix Version/s: 0.18.0
• Hadoop Flags: Reviewed
• Assignee: Hairong Kuang
• Reporter: Koji Noguchi
• Created: 11/Jun/08 19:38
• Updated: 08/Jul/09 16:43
• Resolved: 23/Jun/08 18:01

Description

When we brought up one of our clusters, couple of datanodes failed to start up with this error messages on the log.

2008-06-11 17:25:06,418 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
2008-06-11 17:26:06,748 INFO org.apache.hadoop.dfs.DataNode: Problem connecting to server: namenode/11.111.11.111:8020
2008-06-11 17:27:07,763 INFO org.apache.hadoop.dfs.DataNode: Problem connecting to server: namenode/11.111.11.111:8020
2008-06-11 17:28:08,778 INFO org.apache.hadoop.dfs.DataNode: Problem connecting to server: namenode/11.111.11.111:8020
2008-06-11 17:29:09,793 INFO org.apache.hadoop.dfs.DataNode: Problem connecting to server: namenode/11.111.11.111:8020
2008-06-11 17:29:43,235 ERROR org.apache.hadoop.dfs.DataNode: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1938)
at java.lang.String.substring(String.java:1905)
at org.apache.hadoop.net.NetworkTopology$InnerNode.getNextAncestorName(NetworkTopology.java:119)
at org.apache.hadoop.net.NetworkTopology$InnerNode.remove(NetworkTopology.java:201)
at org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:354)
at org.apache.hadoop.dfs.FSNamesystem.registerDatanode(FSNamesystem.java:2071)
at org.apache.hadoop.dfs.NameNode.register(NameNode.java:536)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

at org.apache.hadoop.ipc.Client.call(Client.java:557)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy4.register(Unknown Source)
at org.apache.hadoop.dfs.DataNode.register(DataNode.java:456)
at org.apache.hadoop.dfs.DataNode.runDatanodeDaemon(DataNode.java:2694)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2729)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:2850)