[HADOOP-3802] IPC - Heartbeat exceptions filling up log files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.17.1
• Component/s: ipc

Environment

Linux 2.6.17-1.2142_FC4smp #1 SMP Tue Jul 11 22:59:20 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux – Fedora Core 4

• Assignee: Unassigned
• Reporter: Dima Brodsky
• Created: 21/Jul/08 17:38
• Updated: 16/Sep/08 17:24
• Resolved: 21/Jul/08 19:53

Description

We have a datanode (10.0.0.93) that is in a semi-live state. An ssh session is able to do a connect but is unable to send or receive any data. The connection is closed immediately after the connection is established. Because of this, the name node's logs are full of the following message:

2008-07-21 09:36:10,800 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1on 9000, call sendHeartbeat(10.0.0.193:50010, 917638877184, 20480, 644100882925, 0, 0) from 10.0.0.193:55908: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

Approximately generating 100MB a minute.

Issue Links

• duplicates: HADOOP-3758 Excessive exceptions in HDFS namenode log file