Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3007 ARM/aarch64 platform support
  3. KUDU-3107

TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service queue is full

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: NA
    • Component/s: None
    • Labels:
      None

      Description

      The test TestRpc.TestCancellationMultiThreads fail sometimes on ARM mechine due the the "service queue full" error. related  error message:

      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 318)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 319)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 320)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 321)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 324)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 332)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 334)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 335)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 336)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 337)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 338)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 339)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 340)
      Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 341)
      F0416 13:01:38.616358 31937 rpc-test.cc:1471] Check failed: controller.status().IsAborted() || controller.status().IsServiceUnavailable() || controller.status().ok() Remote error: Service unavailable: PushStrings request on kudu.rpc.GenericCalculatorService from 127.0.0.1:41516 dropped due to backpressure. The service queue is full; it has 100 items.
      *** Check failure stack trace: ***
      PC: @                0x0 (unknown)
      *** SIGABRT (@0x3e800006bbf) received by PID 27583 (TID 0xffff84b1f050) from PID 27583; stack trace: ***
          @     0xffff93cf0464 raise at ??:0
          @     0xffff93cf18b4 abort at ??:0
          @     0xffff942c5fdc google::logging_fail() at ??:0
          @     0xffff942c7d40 google::LogMessage::Fail() at ??:0
          @     0xffff942c9c78 google::LogMessage::SendToLog() at ??:0
          @     0xffff942c7874 google::LogMessage::Flush() at ??:0
          @     0xffff942ca4fc google::LogMessageFatal::~LogMessageFatal() at ??:0
          @     0xaaaadcee4940 kudu::rpc::SendAndCancelRpcs() at ??:0
          @     0xaaaadcee4b98 _ZZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvENKUlvE_clEv at ??:0
          @     0xaaaadcee76bc _ZSt13__invoke_implIvZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
          @     0xaaaadcee7484 _ZSt8__invokeIZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ at ??:0
          @     0xaaaadcee8208 _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
          @     0xaaaadcee8168 _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEclEv at ??:0
          @     0xaaaadcee8110 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEEE6_M_runEv at ??:0
          @     0xffff93f22e94 (unknown) at ??:0
          @     0xffff93e1e088 start_thread at ??:0
          @     0xffff93d8e4ec (unknown) at ??:0
      

      The attatchment is the full test log

        Attachments

        1. rpc-test.txt
          61 kB
          liusheng

          Activity

            People

            • Assignee:
              RuiChen RuiChen
              Reporter:
              seanlau liusheng
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: