Description
In ipc/Client.java, if a call times out, a SocketTimeoutException is thrown but the Call object still exists on the queue.
What I found was that when transferring very large amounts of data, it's common for queued up calls to timeout. Yet even though the caller has is no longer waiting, the request is still serviced on the server and the data is sent to the client. The client after receiving the full response calls callComplete() which is a noop since nobody is waiting.
The problem is that the calls that timeout will retry and the system gets into a situation where data is being transferred around, but it's all data for timed out requests and no progress is ever made.
My quick solution to this was to add a "boolean timedout" to the Call object which I set to true whenever the queued caller times out. And then when the client starts to pull over the response data (in Connection::run) to first check if the Call is timedout and immediately close the connection.
I think a good fix for this is to queue requests on the client, and do a single sendParam only when there is no outstanding request. This will allow closing the connection when receiving a response for a request we no longer have pending, reopen the connection, and resend the next queued request. I can provide a patch for this, but I've seen a lot of recent activity in this area so I'd like to get some feedback first.
Attachments
Attachments
Issue Links
- relates to
-
HADOOP-572 Chain reaction in a big cluster caused by simultaneous failure of only a few data-nodes.
-
- Closed
-