Critical Description
HBASE-2937 introduces rpc-timeout and sets SO_TIMEOUT parameter of the socket to throw SocketTimeoutException. That means the exception can be thrown from any code that reads data directly/indirectly from the socket. If the exception is thrown in the middle of reading a set of data, it is required to drag out and drop the rest part of the set of data from the socket and make ready to read the next data, in order to reuse the socket. It seems difficult, and I can't find such recovering code in HBaseClient.
I think, when IO streams wrapping the socket throw an exception, the nesting connection instance should be discarded, and rpc-timeout should be handled separately from SO_TIMEOUT parameter.