[KUDU-2075] Crash when using tracing in SetupThreadLocalBuffer

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.8.0
• Component/s: util
• Target Version/s: 1.8.0
• Assignee: Todd Lipcon
• Reporter: Jean-Daniel Cryans
• Created: 21/Jul/17 20:50
• Updated: 22/Mar/18 18:33
• Resolved: 22/Mar/18 18:33

Description

Got this crash while tracing:

F0721 13:14:12.038748  2708 map-util.h:414] Check failed: InsertIfNotPresent(collection, key, data) duplicate key: 139914842822400

Backtrace:

#0  0x000000348aa32625 in raise () from /lib64/libc.so.6
#1  0x000000348aa33e05 in abort () from /lib64/libc.so.6
#2  0x0000000001b53d29 in kudu::AbortFailureFunction () at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/util/minidump.cc:186
#3  0x00000000008b9e1d in google::LogMessage::Fail () at /usr/src/debug/kudu-1.4.0-cdh5.12.0/thirdparty/src/glog-0.3.5/src/logging.cc:1488
#4  0x00000000008bbcdd in google::LogMessage::SendToLog (this=Unhandled dwarf expression opcode 0xf3
) at /usr/src/debug/kudu-1.4.0-cdh5.12.0/thirdparty/src/glog-0.3.5/src/logging.cc:1442
#5  0x00000000008b9959 in google::LogMessage::Flush (this=0x7f40768144f0) at /usr/src/debug/kudu-1.4.0-cdh5.12.0/thirdparty/src/glog-0.3.5/src/logging.cc:1311
#6  0x00000000008bc77f in google::LogMessageFatal::~LogMessageFatal (this=0x7f40768144f0, __in_chrg=<value optimized out>)
    at /usr/src/debug/kudu-1.4.0-cdh5.12.0/thirdparty/src/glog-0.3.5/src/logging.cc:2023
#7  0x0000000001b0915f in InsertOrDie<std::unordered_map<long, kudu::debug::TraceLog::PerThreadInfo*> > (collection=0x36265a8, key=Unhandled dwarf expression opcode 0xf3
)
    at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/gutil/map-util.h:414
#8  0x0000000001b00c18 in kudu::debug::TraceLog::SetupThreadLocalBuffer (this=0x3626300) at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/util/debug/trace_event_impl.cc:1715
#9  0x0000000001b052d8 in kudu::debug::TraceLog::AddTraceEventWithThreadIdAndTimestamp (this=0x3626300, phase=Unhandled dwarf expression opcode 0xf3
)
    at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/util/debug/trace_event_impl.cc:1773
#10 0x0000000000ab616d in AddTraceEventWithThreadIdAndTimestamp<unsigned long> (this=0x59dd5e40, entry_batches=std::vector of length 1, capacity 1 = {...})
    at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/util/debug/trace_event.h:1315
#11 AddTraceEvent<unsigned long> (this=0x59dd5e40, entry_batches=std::vector of length 1, capacity 1 = {...})
    at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/util/debug/trace_event.h:1331
#12 kudu::log::Log::AppendThread::HandleGroup (this=0x59dd5e40, entry_batches=std::vector of length 1, capacity 1 = {...})
    at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/consensus/log.cc:335
#13 0x0000000000ab6707 in kudu::log::Log::AppendThread::DoWork (this=0x59dd5e40) at /usr/src/debug/kudu-1.4.0-cdh5.12.0/src/kudu/consensus/log.cc:326
#14 0x0000000001b8d7d6 in operator() (this=0x21c2a180, permanent=false)