[KUDU-761] TS seg fault after short log append

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: M5
• Fix Version/s: NA
• Component/s: log
• Target Version/s: Backlog
• Assignee: Unassigned
• Reporter: Adar Dembo
• Created: 13/May/15 13:49
• Updated: 29/May/20 19:02
• Resolved: 29/May/20 19:02

Description

I was running tpch_real_world with SF=6000 on a2414. Towards the end of the run, flush MM ops consistently won out over log gc MM due to memory pressure. Eventually we ran out of disk space on the disk hosting the logs:

E0511 05:45:47.913038 8174 log.cc:130] Error appending to the log: IO error: pwritev error: expected to write 254370 bytes, wrote 145525 bytes instead

What followed was a SIGSEGV:

PC: @ 0x0 
*** SIGSEGV (@0x0) received by PID 8004 (TID 0x7fae308fd700) from PID 0; stack trace: ***
@ 0x3d4ae0f500 
@ 0x0 
@ 0x8a2791 kudu::log::LogEntryPB::~LogEntryPB()
@ 0x8a2272 kudu::log::LogEntryBatchPB::~LogEntryBatchPB()
@ 0x8a22e1 kudu::log::LogEntryBatchPB::~LogEntryBatchPB()
@ 0x892de4 kudu::log::Log::AppendThread::RunThread()

There's not much we can do when we run out of disk space, but better to crash with a CHECK or something than to SIGSEGV.

Issue Links

• duplicates: KUDU-2404 Mitigate effects of full disks