[KUDU-2086] Uneven assignment of connections to Reactor threads creates skew and limits transfer throughput

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: perf, rpc
• Assignee: Joe McDonnell
• Reporter: Mostafa Mokhtar
• Created: 31/Jul/17 23:24
• Updated: 02/Jun/20 20:23

Description

Uneven assignment of connections to Reactor threads causes a couple of reactor threads to run @100% which limits overall system throughput.

Increasing the number of reactor threads alleviate the problem but some threads are still running much hotter than others.

Snapshot below is from a 20 node cluster

ps -T -p 69387 | grep rpc |  grep -v "00:00"  | awk '{print $4,$0}' | sort
00:03:17  69387  69596 ?        00:03:17 rpc reactor-695
00:03:20  69387  69632 ?        00:03:20 rpc reactor-696
00:03:21  69387  69607 ?        00:03:21 rpc reactor-696
00:03:25  69387  69629 ?        00:03:25 rpc reactor-696
00:03:26  69387  69594 ?        00:03:26 rpc reactor-695
00:03:34  69387  69595 ?        00:03:34 rpc reactor-695
00:03:35  69387  69625 ?        00:03:35 rpc reactor-696
00:03:38  69387  69570 ?        00:03:38 rpc reactor-695
00:03:38  69387  69620 ?        00:03:38 rpc reactor-696
00:03:47  69387  69639 ?        00:03:47 rpc reactor-696
00:03:48  69387  69593 ?        00:03:48 rpc reactor-695
00:03:49  69387  69591 ?        00:03:49 rpc reactor-695
00:04:04  69387  69600 ?        00:04:04 rpc reactor-696
00:07:16  69387  69640 ?        00:07:16 rpc reactor-696
00:07:39  69387  69616 ?        00:07:39 rpc reactor-696
00:07:54  69387  69572 ?        00:07:54 rpc reactor-695
00:09:10  69387  69613 ?        00:09:10 rpc reactor-696
00:09:28  69387  69567 ?        00:09:28 rpc reactor-695
00:09:39  69387  69603 ?        00:09:39 rpc reactor-696
00:09:42  69387  69641 ?        00:09:42 rpc reactor-696
00:09:59  69387  69604 ?        00:09:59 rpc reactor-696
00:10:06  69387  69623 ?        00:10:06 rpc reactor-696
00:10:43  69387  69636 ?        00:10:43 rpc reactor-696
00:10:59  69387  69642 ?        00:10:59 rpc reactor-696
00:11:28  69387  69585 ?        00:11:28 rpc reactor-695
00:12:43  69387  69598 ?        00:12:43 rpc reactor-695
00:15:42  69387  69578 ?        00:15:42 rpc reactor-695
00:16:10  69387  69614 ?        00:16:10 rpc reactor-696
00:17:43  69387  69575 ?        00:17:43 rpc reactor-695

Issue Links

• blocks: IMPALA-5131 Known issues with KRPC (IMPALA-2567)
• is related to: IMPALA-2567 KRPC milestone 1
• relates to: IMPALA-5865 Improve Impala execution scalability