[KUDU-1466] C++ client errors misreported as GetTableLocations timeouts

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.8.0
• Component/s: client
• Target Version/s: 1.8.0
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 24/May/16 06:13
• Updated: 08/Jul/21 05:40

Description

client-test is currently very flaky due to this issue:

• we are injecting some kind of failure on the tablet server (eg DNS resolution failure)
• when we fail to connect to the TS, we correctly re-trigger a lookup against the master
• depending how the backoffs and retries line up, we sometimes end up triggering the lookup retry when the remaining operation budget is very short (eg <10ms)
  • this GetTabletLocations RPC times out since the master is unable to respond within the ridiculously short timeout

During the course of retrying some operation, we should probably not replace the 'last_error' with a master error, so long as we have had at least one successful master lookup (thus indicating that the master is not the problem)

Issue Links

• is duplicated by: KUDU-1552 ClientTest.TestFailedDnsResolution is flaky
• links to: The scope document on addressing this issue