[HBASE-10432] Rpc retries non-recoverable error

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.98.0, 0.96.2, 0.99.0
• Fix Version/s: 0.96.2, 0.98.1, 0.99.0
• Component/s: IPC/RPC
• Hadoop Flags: Reviewed
• Assignee: Nick Dimiduk
• Reporter: Nick Dimiduk
• Created: 28/Jan/14 17:55
• Updated: 21/Feb/15 23:32
• Resolved: 06/Mar/14 17:46

Description

I'm recently working with hbase/trunk + hive/trunk. I had a hive command eventually timeout with the following exception (stacktrace truncated).

Caused by: java.io.IOException: Could not set up IO Streams
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:922)
        at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1536)
        at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1425)
        at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
        at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:28857)
        at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:302)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
        ... 43 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
        at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:861)
        ... 52 more

The root cause looks like a dependency version missmatch (Hive compiled vs hadoop1, HBase vs hadoop2). However, we still retry this exception, even though it'll never actually complete. We should be more careful where we blindly catch Throwables.