[HADOOP-375] Introduce a way for datanodes to register their HTTP info ports with the NameNode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5.0
• Fix Version/s: 0.5.0
• Assignee: Devaraj Das
• Reporter: Devaraj Das
• Created: 21/Jul/06 10:48
• Updated: 08/Jul/09 16:41
• Resolved: 26/Jul/06 12:13

Description

If we have multiple datanodes within a single machine the Jetty servers (other than the first one) won't be able to bind to the fixed HTTP port. So, one solution is to have the datanodes pick a free port (starting from a configured port value) and then inform namenode about it so that the namenode can then do redirects, etc.

Johan Oskarson reported this problem.

If a computer have a second dfs data dir in the config it doesn't start properly because of:

Exception in thread "main" java.io.IOException: Problem starting http server
at org.apache.hadoop.mapred.StatusHttpServer.start(StatusHttpServer.java:182)
at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:170)
at org.apache.hadoop.dfs.DataNode.makeInstanceForDir(DataNode.java:1045)
at org.apache.hadoop.dfs.DataNode.run(DataNode.java:999)
at org.apache.hadoop.dfs.DataNode.runAndWait(DataNode.java:1015)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1066)
Caused by: org.mortbay.util.MultiException[java.net.BindException: Address already in use]
at org.mortbay.http.HttpServer.doStart(HttpServer.java:731)
at org.mortbay.util.Container.start(Container.java:72)
at org.apache.hadoop.mapred.StatusHttpServer.start(StatusHttpServer.java:159)
... 5 more

Issue Links

• incorporates: HADOOP-376 Datanode does not scan for an open http port