[TS-970] ts crash when use evacuate feature

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.1
• Component/s: Cache

Environment

# uname -a
Linux cache170.cn63 2.6.32-131.4.1.tb204.el5.x86_64 #1 SMP Fri Jul 1 22:17:55 CST 2011 x86_64 x86_64 x86_64 GNU/Linux

• Assignee: mohan_zl
• Reporter: mohan_zl
• Due: 27/Sep/11
• Created: 27/Sep/11 10:28
• Updated: 09/Nov/11 10:05
• Resolved: 09/Nov/11 10:05

Description

My config arguments:
start trafficserver in one machine 1

# traffic_line -s "proxy.config.cache.hit_evacuate_percent" -v 5   // by default, the value is 0, means no evacuation
# cat storage.config  //the size of storage file is 5G
/dev/sda1                                                          

My test method:
use http_load from another machine 2 to test machine 1

# http_load -parallel 500 -fetches 10000000 -proxy xxx.xx.xx.xxx:8080 mohan_urls
# wc -l mohan_urls
40849490 mohan_urls
# sort mohan_urls | uniq -c > mohan_urls_1; wc -l mohan_urls_1
14057051 mohan_urls_1
# ls -al
-rw-r--r-- 1 root root   3253681200 Sep 21 20:42 mohan_urls
-rw-r--r-- 1 root root   1305299567 Sep 21 21:27 mohan_urls_1
The total access urls' size is larger than the storage size

Crash result:

(gdb) bt
#0  0x0000003639c30265 in raise () from /lib64/libc.so.6
#1  0x0000003639c31d10 in abort () from /lib64/libc.so.6
#2  0x00002aaaee6d86fa in ink_die_die_die (retval=1) at ink_error.cc:43
#3  0x00002aaaee6d8979 in ink_fatal_va (return_code=1, message_format=0x424e6700 "CacheWrite.cc:519: failed assert `dir_pinned(&dir)`", 
    ap=0x424e6600) at ink_error.cc:65
#4  0x00002aaaee6d8b46 in ink_fatal (return_code=1, message_format=0x424e6700 "CacheWrite.cc:519: failed assert `dir_pinned(&dir)`")
    at ink_error.cc:73
#5  0x00002aaaee6d697a in _ink_assert (a=0x767108 "dir_pinned(&dir)", f=0x766a46 "CacheWrite.cc", l=519) at ink_assert.cc:44
#6  0x00000000006afaa2 in CacheVC::evacuateDocDone (this=0x2aaaac371f80, event=3900, e=0x0) at CacheWrite.cc:519
#7  0x00000000004d3789 in Continuation::handleEvent (this=0x2aaaac371f80, event=3900, data=0x0) at I_Continuation.h:146
#8  0x00000000006b0e7b in Vol::aggWrite (this=0x2e18a10, event=3900, e=0x2e18ae0) at CacheWrite.cc:998
#9  0x00000000006b1553 in Vol::evacuateWrite (this=0x2e18a10, evacuator=0x2aaaac371f80, event=3900, e=0x2e18ae0) at CacheWrite.cc:603
#10 0x00000000006b2031 in Vol::evacuateDocReadDone (this=0x2e18a10, event=3900, e=0x2e18ae0) at CacheWrite.cc:687
#11 0x00000000004d3789 in Continuation::handleEvent (this=0x2e18a10, event=3900, data=0x2e18ae0) at I_Continuation.h:146
#12 0x000000000068a532 in AIOCallbackInternal::io_complete (this=0x2e18ae0, event=1, data=0x30687d0) at P_AIO.h:80
#13 0x00000000004d3789 in Continuation::handleEvent (this=0x2e18ae0, event=1, data=0x30687d0) at I_Continuation.h:146
#14 0x00000000006f69b0 in EThread::process_event (this=0x2aaaab22d010, e=0x30687d0, calling_code=1) at UnixEThread.cc:142
#15 0x00000000006f6cf2 in EThread::execute (this=0x2aaaab22d010) at UnixEThread.cc:219
#16 0x00000000006f6337 in spawn_thread_internal (a=0x2d75190) at Thread.cc:88
#17 0x000000363a8064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003639cd3c2d in clone () from /lib64/libc.so.6

(gdb) f 6
#6  0x00000000006afaa2 in CacheVC::evacuateDocDone (this=0x2aaaac371f80, event=3900, e=0x0) at CacheWrite.cc:519
519                 ink_assert(dir_pinned(&dir));
(gdb) p cod->num_writers
$1 = 1
(gdb) p cod->max_writers
$2 = 5
(gdb) p cod->dont_update_directory
$3 = false
(gdb) p cod->move_resident_alt
$4 = false
(gdb) p cod->reading_vec
$5 = false
(gdb) p cod->writing_vec
$6 = true
(gdb) p cod->first_dir
$7 = {w = {61517, 5121, 14401, 59410, 0}}
(gdb) p cod->single_doc_dir
$8 = {w = {61517, 5121, 12585, 59410, 0}}
(gdb) p this->dir
$9 = {w = {10203, 5167, 14401, 59409, 0}}
(gdb) p this->overwrite_dir
$10 = {w = {20152, 5167, 10305, 59409, 0}}