[TS-1080] Assert under heavy load with logging enabled

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.1.4
• Component/s: Logging
• Assignee: Leif Hedstrom
• Reporter: Leif Hedstrom
• Created: 17/Jan/12 21:26
• Updated: 06/Jun/12 00:01
• Resolved: 31/Mar/12 04:36

Description

Given enough load (in the 100,000 QPS or more), we run out of some sort of buffer space, with an assert of

#0  0x00002ba719d50285 in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00002ba719d51b9b in __GI_abort () at abort.c:91
#2  0x00000000006b561a in ink_die_die_die (retval=<optimized out>) at ink_error.cc:43
#3  ink_fatal_va(int, const char *, typedef __va_list_tag __va_list_tag *) (return_code=<optimized out>, message_format=<optimized out>, ap=0x7fff7275a7d8) at ink_error.cc:65
#4  0x00000000006b56a7 in ink_fatal (return_code=<optimized out>, message_format=<optimized out>) at ink_error.cc:73
#5  0x00000000006b4970 in _ink_assert (a=0x6fd380 "_num_flush_buffers[_open_flush_array] < FLUSH_ARRAY_SIZE", f=<optimized out>, l=96) at ink_assert.cc:44
#6  0x00000000005a8b34 in add_to_flush_queue (buffer=0x2ba7443ca970, this=0x22fb918) at LogObject.h:96
#7  LogObject::_checkout_write (this=0x22fb880, write_offset=0x7fff7275add8, bytes_needed=152) at LogObject.cc:455
#8  0x00000000005a8fd3 in LogObject::log (this=0x22fb880, lad=0x7fff7275b030, text_entry=0x0) at LogObject.cc:580
#9  0x000000000058e956 in log (lad=0x7fff7275b030, this=<optimized out>) at LogObject.h:475
#10 Log::access (lad=0x7fff7275b030) at Log.cc:1086

Increasing FLUSH_ARRAY_SIZE alleviates the problem, but really, we shouldn't end up in this situation at all.