Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2086

Uneven assignment of connections to Reactor threads creates skew and limits transfer throughput

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: rpc
    • Labels:
      None

      Description

      Uneven assignment of connections to Reactor threads causes a couple of reactor threads to run @100% which limits overall system throughput.

      Increasing the number of reactor threads alleviate the problem but some threads are still running much hotter than others.

      Snapshot below is from a 20 node cluster

      ps -T -p 69387 | grep rpc |  grep -v "00:00"  | awk '{print $4,$0}' | sort
      00:03:17  69387  69596 ?        00:03:17 rpc reactor-695
      00:03:20  69387  69632 ?        00:03:20 rpc reactor-696
      00:03:21  69387  69607 ?        00:03:21 rpc reactor-696
      00:03:25  69387  69629 ?        00:03:25 rpc reactor-696
      00:03:26  69387  69594 ?        00:03:26 rpc reactor-695
      00:03:34  69387  69595 ?        00:03:34 rpc reactor-695
      00:03:35  69387  69625 ?        00:03:35 rpc reactor-696
      00:03:38  69387  69570 ?        00:03:38 rpc reactor-695
      00:03:38  69387  69620 ?        00:03:38 rpc reactor-696
      00:03:47  69387  69639 ?        00:03:47 rpc reactor-696
      00:03:48  69387  69593 ?        00:03:48 rpc reactor-695
      00:03:49  69387  69591 ?        00:03:49 rpc reactor-695
      00:04:04  69387  69600 ?        00:04:04 rpc reactor-696
      00:07:16  69387  69640 ?        00:07:16 rpc reactor-696
      00:07:39  69387  69616 ?        00:07:39 rpc reactor-696
      00:07:54  69387  69572 ?        00:07:54 rpc reactor-695
      00:09:10  69387  69613 ?        00:09:10 rpc reactor-696
      00:09:28  69387  69567 ?        00:09:28 rpc reactor-695
      00:09:39  69387  69603 ?        00:09:39 rpc reactor-696
      00:09:42  69387  69641 ?        00:09:42 rpc reactor-696
      00:09:59  69387  69604 ?        00:09:59 rpc reactor-696
      00:10:06  69387  69623 ?        00:10:06 rpc reactor-696
      00:10:43  69387  69636 ?        00:10:43 rpc reactor-696
      00:10:59  69387  69642 ?        00:10:59 rpc reactor-696
      00:11:28  69387  69585 ?        00:11:28 rpc reactor-695
      00:12:43  69387  69598 ?        00:12:43 rpc reactor-695
      00:15:42  69387  69578 ?        00:15:42 rpc reactor-695
      00:16:10  69387  69614 ?        00:16:10 rpc reactor-696
      00:17:43  69387  69575 ?        00:17:43 rpc reactor-695
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joemcdonnell Joe McDonnell
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: