[KUDU-1802] Deserializing scan tokens should avoid round-trip to master

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.13.0
• Component/s: client, perf
• Labels: impala ramp-up
• Assignee: Grant Henke
• Reporter: Todd Lipcon
• Created: 12/Dec/16 03:12
• Updated: 30/Aug/22 01:26
• Resolved: 07/Jul/20 14:23

Description

Currently, KuduScanToken::DeserializeIntoScanner calls KuduClient::OpenTable() which makes a GetTableSchema call to the master. This round trip is a bit expensive because it's always a "thundering herd" for an Impala query or Spark job – every host deserializes a bunch of scan tokens at the same time and ends up having to back off.

We should consider some ways to avoid this.

Issue Links

• causes: KUDU-3254 Crash in Kudu C++ client when working with stale scan tokens containing tablet location info
• causes: KUDU-3349 Kudu java client failed to demote leader and caused a lot of deleting rows timeout
• is related to: KUDU-3115 Improve scalability of Kudu masters
• is related to: KUDU-3135 Add Client Metadata Tokens
• is related to: KUDU-3146 Consistent table metadata for scans
• is related to: IMPALA-9903 Queries on a Kudu table call openTable multiple times