[IMPALA-9792] Split Kudu scan ranges into smaller chunks for greater paralellelism

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend
• Labels: kudu multithreading
• Epic Link: Multithreading upgrade path for large clusters
• Epic Color: ghx-label-8
• Assignee: Bikramjeet Vig
• Reporter: Tim Armstrong
• Created: 28/May/20 22:19
• Updated: 08/Feb/21 17:19

Description

We currently use one thread to scan each tablet, which may underparallelise queries in many cases. Kudu added an API in KUDU-2437 and KUDU-2670 to split tokens at a finer granularity.

See
https://github.com/apache/kudu/commit/22a6faa44364dec3a171ec79c15b814ad9277d8f#diff-a4afa9dba99c7612b2cb9176134ff2b0

The major downside is that the planner has to do an extra RPC to a tserver for each tablet being scanned in order to figure out key range splits. Maybe we can tie this to mt_dop >= 2, or use some heuristics to avoid these RPCs for smaller tables.

Issue Links

• causes: IMPALA-10245 Test fails in TestKuduReadTokenSplit.test_kudu_scanner
• is related to: KUDU-2670 Splitting more tasks for spark job, and add more concurrent for scan operation
• is related to: KUDU-2437 Split a tablet into primary key ranges by size
• relates to: IMPALA-9656 Dynamic intra-node load balancing for Kudu (and maybe HBase) scans.