[HADOOP-1739] ConnectException in TaskTracker Child

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.15.0

Environment

Version: 0.15.0-dev, r565628
Compiled: Tue Aug 14 20:55:37 UTC 2007 by hadoopqa
1400 nodes

• Assignee: Doug Cutting
• Reporter: Srikanth Kakani
• Created: 20/Aug/07 22:29
• Updated: 08/Jul/09 16:52
• Resolved: 30/Aug/07 18:49

Description

Steps to Reproduce:
I had 11000 mappers and 2700 reducers in a job and most failures correspond to the following logs:

Stderr:
Exception in thread "main" java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:193)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:150)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:530)
at org.apache.hadoop.ipc.Client.call(Client.java:459)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:165)
at org.apache.hadoop.mapred.$Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:248)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1781)

Syslog:
2007-08-19 18:45:07,490 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 1 time(s).
2007-08-19 18:45:08,494 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 2 time(s).
2007-08-19 18:45:09,497 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 3 time(s).
2007-08-19 18:45:10,500 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 4 time(s).
2007-08-19 18:45:11,503 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 5 time(s).
2007-08-19 18:45:12,506 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 6 time(s).
2007-08-19 18:45:13,508 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 7 time(s).
2007-08-19 18:45:14,511 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 8 time(s).
2007-08-19 18:45:15,512 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 9 time(s).
2007-08-19 18:45:16,515 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:50051. Already tried 10 time(s)