[KUDU-1340] Lock order inversion between RpcController and OutboundCall

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.8.0
• Component/s: rpc
• Target Version/s: 0.8.0
• Code Review: http://gerrit.cloudera.org:8080/2240
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 19/Feb/16 02:51
• Updated: 19/Feb/16 04:46
• Resolved: 19/Feb/16 04:46

Description

I enabled deadlock detection in TSAN today and it pointed out the following, which looks legit:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=4232)
  Cycle in lock order graph: M11532 (0x7d440006dd88) => M10486 (0x7d4c00053fb0) => M11532

  Mutex M10486 acquired here while holding mutex M11532 in thread T6:
    #3 kudu::lock_guard<kudu::simple_spinlock>::lock_guard(kudu::simple_spinlock*) /home/todd/git/kudu/build/tsan/../../src/kudu/util/locks.h:241:5 (libtserver.so+0x000000130339)
    #4 kudu::rpc::RpcController::timeout() const /home/todd/git/kudu/build/tsan/../../src/kudu/rpc/rpc_controller.cc:91:31 (libkrpc.so+0x0000000d4506)
    #5 kudu::rpc::OutboundCall::SetTimedOut() /home/todd/git/kudu/build/tsan/../../src/kudu/rpc/outbound_call.cc:262:9 (libkrpc.so+0x00000009d825)

  Mutex M11532 acquired here while holding mutex M10486 in thread T132:
    #3 kudu::lock_guard<kudu::simple_spinlock>::lock_guard(kudu::simple_spinlock*) /home/todd/git/kudu/build/tsan/../../src/kudu/util/locks.h:241:5 (libtserver.so+0x000000130339)
    #4 kudu::rpc::OutboundCall::IsFinished() const /home/todd/git/kudu/build/tsan/../../src/kudu/rpc/outbound_call.cc:274:31 (libkrpc.so+0x00000009c785)
    #5 kudu::rpc::RpcController::finished() const /home/todd/git/kudu/build/tsan/../../src/kudu/rpc/rpc_controller.cc:57:12 (libkrpc.so+0x0000000d41d1)

(trimmed to fit in JIRA better)