[KUDU-1656] Scanner timeouts aren't retried when waiting on a transaction

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: tserver
• Target Version/s: 1.2.0
• Assignee: David Alves
• Reporter: Jean-Daniel Cryans
• Created: 27/Sep/16 22:05
• Updated: 09/Dec/16 04:44
• Resolved: 09/Dec/16 04:44

Description

I recently changed ITClient to use READ_AT_SNAPSHOT scanners and we've been seeing errors like this:

19:56:29.459 [WARN - New I/O worker #169] (AsyncKuduScanner.java:407) Can not open scanner
org.apache.kudu.client.NonRecoverableException: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for all transactions with ts < P: 1475006188645381 usec, L: 0 to commit
	at org.apache.kudu.client.TabletClient.dispatchTSErrorOrReturnException(TabletClient.java:548)
	at org.apache.kudu.client.TabletClient.decode(TabletClient.java:482)
	at org.apache.kudu.client.TabletClient.decode(TabletClient.java:83)

Since this comes back as a TimedOut AppStatus, neither clients are retrying the error which doesn't seem to be the expected behavior on the server-side: https://github.com/cloudera/kudu/blob/be719edc3581802e094c3af6a88d67acba44ba71/src/kudu/tserver/tablet_service.cc#L1764

One one hand it seems weird to rely on the user to retry only certain timeouts, OTOH maybe it shouldn't be sent as a timeout? But I'm not sure what it should be.

Issue Links

• Is contained by: KUDU-1703 Handle snapshot reads that might block indefinitely
• is related to: KUDU-1753 Impala query fails: Unable to advance iterator: Illegal state: Tablet is not running
• is related to: KUDU-430 Consistent Operations