[KUDU-3295] kudu client cannot read if one of tablet replica is unhealthy

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.0, 1.12.0, 1.11.1, 1.13.0, 1.14.0, 1.15.0
• Fix Version/s: 1.16.0
• Component/s: client, java
• Code Review: https://gerrit.cloudera.org/#/c/17971/
• Assignee: Unassigned
• Reporter: Redriver
• Created: 17/Jun/21 11:02
• Updated: 08/Feb/22 08:51
• Resolved: 27/Oct/21 05:31

Description

I found this issue when one of my tserver was down, as a result, DNS resolver failed and the tablet replicas on that server cannot access. When I use kudu java API to scan the table, the client complains. That is unexpected for only one tserver crashed which blocks the read.

the number of replicas does not equal the number of servers

Caused by: java.lang.IllegalArgumentException: the number of replicas does not equal the number of servers
    at org.apache.kudu.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
    at org.apache.kudu.client.RemoteTablet.<init>(RemoteTablet.java:84)
    at org.apache.kudu.client.AsyncKuduClient.discoverTablets(AsyncKuduClient.java:2414)
    at org.apache.kudu.client.AsyncKuduClient$MasterLookupCB.call(AsyncKuduClient.java:2301)
    at org.apache.kudu.client.AsyncKuduClient$MasterLookupCB.call(AsyncKuduClient.java:2282)
    at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
    at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
    at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
    at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
    at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:324)
    at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:321)
    at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
    at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
    at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
    at org.apache.kudu.client.Connection.channelRead0(Connection.java:362)
...