[TS-3775] ASAN crash while running regression test Cache_vol

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Cache
• Assignee: Susan Hinrichs
• Reporter: Susan Hinrichs
• Created: 17/Jul/15 11:54
• Updated: 09/Nov/15 21:02
• Resolved: 18/Jul/15 13:05

Description

Seen while running master built with ASAN on FC 21. I have a patch which I'll attach and discuss in comment.

REGRESSION TEST Cache_vol started
RPRINT Cache_vol: 1 128 Megabyte Volumes
RPRINT Cache_vol: Not enough space for 10 volume
RPRINT Cache_vol: Random Volumes after clearing the disks
RPRINT Cache_vol: volume=1 scheme=http size=128
RPRINT Cache_vol: Random Volumes without clearing the disks
RPRINT Cache_vol: volume=1 scheme=rtsp size=128
=================================================================
==4513==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400008e9e0 at pc 0x989546 bp 0x7fffef2a59b0 sp 0x7fffef2a59a0
READ of size 8 at 0x60400008e9e0 thread T2 ([ET_NET 1])
    #0 0x989545 in cplist_update /home/shinrich/ats/iocore/cache/Cache.cc:2702
    #1 0x989545 in cplist_reconfigure() /home/shinrich/ats/iocore/cache/Cache.cc:2846
    #2 0x9d1186 in execute_and_verify(RegressionTest*) /home/shinrich/ats/iocore/cache/CacheHosting.cc:996
    #3 0x9d2229 in RegressionTest_Cache_vol(RegressionTest*, int, int*) /home/shinrich/ats/iocore/cache/CacheHosting.cc:842
    #4 0x7ffff6cb55f1 in start_test /home/shinrich/ats/lib/ts/Regression.cc:78
    #5 0x7ffff6cb55f1 in RegressionTest::run_some() /home/shinrich/ats/lib/ts/Regression.cc:126
    #6 0x7ffff6cb5b00 in RegressionTest::check_status() /home/shinrich/ats/lib/ts/Regression.cc:141
    #7 0x5404fb in RegressionCont::mainEvent(int, Event*) /home/shinrich/ats/proxy/Main.cc:1210
    #8 0xb6b771 in Continuation::handleEvent(int, void*) /home/shinrich/ats/iocore/eventsystem/I_Continuation.h:146
    #9 0xb6b771 in EThread::process_event(Event*, int) /home/shinrich/ats/iocore/eventsystem/UnixEThread.cc:128
    #10 0xb6d3a6 in EThread::execute() /home/shinrich/ats/iocore/eventsystem/UnixEThread.cc:207
    #11 0xb69da1 in spawn_thread_internal /home/shinrich/ats/iocore/eventsystem/Thread.cc:86
    #12 0x7ffff5e27529 in start_thread (/lib64/libpthread.so.0+0x7529)
    #13 0x7ffff464922c in __clone (/lib64/libc.so.6+0x10022c)

0x60400008e9e0 is located 16 bytes inside of 40-byte region [0x60400008e9d0,0x60400008e9f8)
freed by thread T2 ([ET_NET 1]) here:
    #0 0x7ffff6f5764f in operator delete(void*) (/lib64/libasan.so.1+0x5864f)
    #1 0x9c84ac in CacheDisk::delete_volume(int) /home/shinrich/ats/iocore/cache/CacheDisk.cc:330
    #2 0x989455 in cplist_update /home/shinrich/ats/iocore/cache/Cache.cc:2684
    #3 0x989455 in cplist_reconfigure() /home/shinrich/ats/iocore/cache/Cache.cc:2846
    #4 0x9d1186 in execute_and_verify(RegressionTest*) /home/shinrich/ats/iocore/cache/CacheHosting.cc:996
    #5 0x9d2229 in RegressionTest_Cache_vol(RegressionTest*, int, int*) /home/shinrich/ats/iocore/cache/CacheHosting.cc:842
    #6 0x7ffff6cb55f1 in start_test /home/shinrich/ats/lib/ts/Regression.cc:78
    #7 0x7ffff6cb55f1 in RegressionTest::run_some() /home/shinrich/ats/lib/ts/Regression.cc:126
    #8 0x7ffff6cb5b00 in RegressionTest::check_status() /home/shinrich/ats/lib/ts/Regression.cc:141
    #9 0x5404fb in RegressionCont::mainEvent(int, Event*) /home/shinrich/ats/proxy/Main.cc:1210
    #10 0xb6b771 in Continuation::handleEvent(int, void*) /home/shinrich/ats/iocore/eventsystem/I_Continuation.h:146
    #11 0xb6b771 in EThread::process_event(Event*, int) /home/shinrich/ats/iocore/eventsystem/UnixEThread.cc:128
    #12 0xb6d3a6 in EThread::execute() /home/shinrich/ats/iocore/eventsystem/UnixEThread.cc:207
    #13 0xb69da1 in spawn_thread_internal /home/shinrich/ats/iocore/eventsystem/Thread.cc:86
    #14 0x7ffff5e27529 in start_thread (/lib64/libpthread.so.0+0x7529)

previously allocated by thread T2 ([ET_NET 1]) here:
    #0 0x7ffff6f5714f in operator new(unsigned long) (/lib64/libasan.so.1+0x5814f)
    #1 0x9c770d in CacheDisk::create_volume(int, long, int) /home/shinrich/ats/iocore/cache/CacheDisk.cc:296
    #2 0x98347e in create_volume /home/shinrich/ats/iocore/cache/Cache.cc:3023
    #3 0x989b41 in cplist_reconfigure() /home/shinrich/ats/iocore/cache/Cache.cc:2877
    #4 0x9d1186 in execute_and_verify(RegressionTest*) /home/shinrich/ats/iocore/cache/CacheHosting.cc:996
    #5 0x9d2229 in RegressionTest_Cache_vol(RegressionTest*, int, int*) /home/shinrich/ats/iocore/cache/CacheHosting.cc:842
    #6 0x7ffff6cb55f1 in start_test /home/shinrich/ats/lib/ts/Regression.cc:78
    #7 0x7ffff6cb55f1 in RegressionTest::run_some() /home/shinrich/ats/lib/ts/Regression.cc:126
    #8 0x7ffff6cb5b00 in RegressionTest::check_status() /home/shinrich/ats/lib/ts/Regression.cc:141
    #9 0x5404fb in RegressionCont::mainEvent(int, Event*) /home/shinrich/ats/proxy/Main.cc:1210
    #10 0xb6b771 in Continuation::handleEvent(int, void*) /home/shinrich/ats/iocore/eventsystem/I_Continuation.h:146
    #11 0xb6b771 in EThread::process_event(Event*, int) /home/shinrich/ats/iocore/eventsystem/UnixEThread.cc:128
    #12 0xb6d3a6 in EThread::execute() /home/shinrich/ats/iocore/eventsystem/UnixEThread.cc:207
    #13 0xb69da1 in spawn_thread_internal /home/shinrich/ats/iocore/eventsystem/Thread.cc:86
    #14 0x7ffff5e27529 in start_thread (/lib64/libpthread.so.0+0x7529)

Thread T2 ([ET_NET 1]) created by T0 ([ET_NET 0]) here:
    #0 0x7ffff6f22dba in pthread_create (/lib64/libasan.so.1+0x23dba)
    #1 0xb6aa88 in ink_thread_create ../../lib/ts/ink_thread.h:150
    #2 0xb6aa88 in Thread::start(char const*, unsigned long, void* (*)(void*), void*) /home/shinrich/ats/iocore/eventsystem/Thread.cc:101
    #3 0xb721a1 in EventProcessor::start(int, unsigned long) /home/shinrich/ats/iocore/eventsystem/UnixEventProcessor.cc:140
    #4 0x53dc23 in main /home/shinrich/ats/proxy/Main.cc:1624
    #5 0x7ffff4568fdf in __libc_start_main (/lib64/libc.so.6+0x1ffdf)

SUMMARY: AddressSanitizer: heap-use-after-free /home/shinrich/ats/iocore/cache/Cache.cc:2702 cplist_update
Shadow bytes around the buggy address:
  0x0c0880009ce0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0880009cf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0880009d00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0880009d10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0880009d20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c0880009d30: fa fa 00 00 00 00 00 00 fa fa fd fd[fd]fd fd fa
  0x0c0880009d40: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 fa
  0x0c0880009d50: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
  0x0c0880009d60: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
  0x0c0880009d70: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
  0x0c0880009d80: fa fa 00 00 00 00 00 05 fa fa 00 00 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Contiguous container OOB:fc
  ASan internal:           fe
==4513==ABORTING

Issue Links

• duplicates: TS-3654 ASAN heap-use-after-free in cache-hosting (regression)
• is related to: TS-4004 ASAN crash while running regression test Cache_vol