[KUDU-695] contention on glog disk can block reactor threads

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Public beta
• Fix Version/s: 1.2.0
• Component/s: perf
• Target Version/s: 1.2.0
• Code Review: https://gerrit.cloudera.org/#/c/5321/
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 08/Apr/15 00:30
• Updated: 05/Dec/16 18:42
• Resolved: 05/Dec/16 18:42

Description

After adding a watchdog which shows the stack trace of blocked reactor callbacks, I see that in many cases it's actually due to blocking within glog itself. We should see whether we can use an asynchronous sink in glog to avoid this, or else we need to be more careful to move even more work off of reactors.

Issue Links

• is related to: KUDU-1777 glog drop_log_memory causes a lot of IO to log dir
• relates to: KUDU-693 UpdateConsensus and RequestConsensusVote RPC callbacks block reactor