I recently changed ITClient to use READ_AT_SNAPSHOT scanners and we've been seeing errors like this:
19:56:29.459 [WARN - New I/O worker #169] (AsyncKuduScanner.java:407) Can not open scanner org.apache.kudu.client.NonRecoverableException: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for all transactions with ts < P: 1475006188645381 usec, L: 0 to commit at org.apache.kudu.client.TabletClient.dispatchTSErrorOrReturnException(TabletClient.java:548) at org.apache.kudu.client.TabletClient.decode(TabletClient.java:482) at org.apache.kudu.client.TabletClient.decode(TabletClient.java:83)
Since this comes back as a TimedOut AppStatus, neither clients are retrying the error which doesn't seem to be the expected behavior on the server-side: https://github.com/cloudera/kudu/blob/be719edc3581802e094c3af6a88d67acba44ba71/src/kudu/tserver/tablet_service.cc#L1764
One one hand it seems weird to rely on the user to retry only certain timeouts, OTOH maybe it shouldn't be sent as a timeout? But I'm not sure what it should be.