[KUDU-3107] TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service queue is full

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Fix Version/s: NA
• Assignee: RuiChen
• Reporter: liusheng
• Created: 17/Apr/20 02:28
• Updated: 29/Jul/20 02:08
• Resolved: 20/May/20 17:06

Description

The test TestRpc.TestCancellationMultiThreads fail sometimes on ARM mechine due the the "service queue full" error. related  error message:

Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 318)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 319)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 320)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 321)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 324)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 332)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 334)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 335)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 336)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 337)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 338)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 339)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 340)
Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 (request call id 341)
F0416 13:01:38.616358 31937 rpc-test.cc:1471] Check failed: controller.status().IsAborted() || controller.status().IsServiceUnavailable() || controller.status().ok() Remote error: Service unavailable: PushStrings request on kudu.rpc.GenericCalculatorService from 127.0.0.1:41516 dropped due to backpressure. The service queue is full; it has 100 items.
*** Check failure stack trace: ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e800006bbf) received by PID 27583 (TID 0xffff84b1f050) from PID 27583; stack trace: ***
    @     0xffff93cf0464 raise at ??:0
    @     0xffff93cf18b4 abort at ??:0
    @     0xffff942c5fdc google::logging_fail() at ??:0
    @     0xffff942c7d40 google::LogMessage::Fail() at ??:0
    @     0xffff942c9c78 google::LogMessage::SendToLog() at ??:0
    @     0xffff942c7874 google::LogMessage::Flush() at ??:0
    @     0xffff942ca4fc google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0xaaaadcee4940 kudu::rpc::SendAndCancelRpcs() at ??:0
    @     0xaaaadcee4b98 _ZZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvENKUlvE_clEv at ??:0
    @     0xaaaadcee76bc _ZSt13__invoke_implIvZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
    @     0xaaaadcee7484 _ZSt8__invokeIZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ at ??:0
    @     0xaaaadcee8208 _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
    @     0xaaaadcee8168 _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEclEv at ??:0
    @     0xaaaadcee8110 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEEE6_M_runEv at ??:0
    @     0xffff93f22e94 (unknown) at ??:0
    @     0xffff93e1e088 start_thread at ??:0
    @     0xffff93d8e4ec (unknown) at ??:0

The attatchment is the full test log