Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.8
    • Component/s: None
    • Labels:
      None

      Description

      Opening another bug for this, it can still happen regardless of ccache on or off.

      My setup is fairly simple, mostly standard configs, but setup as a forward proxy. When pointing my browser to use ATS as the proxy, and I go to search.google.com and start typing in the search box, we sometimes segfault.

      (gdb) bt
      #0  0x0000003f2e60e1fd in write () from /lib64/libpthread.so.0
      #1  0x00000000006102d8 in write (this=0x7fffe000bd70, towrite=1260, wattempted=@0x7ffff6c11c78, total_wrote=@0x7ffff6c11c80, 
          buf=<value optimized out>) at ../../iocore/eventsystem/P_UnixSocketManager.h:207
      #2  UnixNetVConnection::load_buffer_and_write (this=0x7fffe000bd70, towrite=1260, wattempted=@0x7ffff6c11c78, 
          total_wrote=@0x7ffff6c11c80, buf=<value optimized out>) at UnixNetVConnection.cc:833
      #3  0x000000000061443b in write_to_net_io (nh=0x7ffff6d15628, vc=0x7fffe000bd70, thread=0x7ffff6d14010)
          at UnixNetVConnection.cc:439
      #4  0x000000000060c42a in NetHandler::mainNetEvent (this=0x7ffff6d15628, event=<value optimized out>, e=<value optimized out>)
          at UnixNet.cc:419
      #5  0x00000000006325e4 in handleEvent (this=0x7ffff6d14010, e=0xe846a0, calling_code=5) at I_Continuation.h:146
      #6  EThread::process_event (this=0x7ffff6d14010, e=0xe846a0, calling_code=5) at UnixEThread.cc:140
      #7  0x0000000000632f73 in EThread::execute (this=0x7ffff6d14010) at UnixEThread.cc:262
      #8  0x000000000063142a in spawn_thread_internal (a=0xe770f0) at Thread.cc:85
      #9  0x0000003f2e6068e0 in start_thread () from /lib64/libpthread.so.0
      #10 0x0000003f2dee0c9d in clone () from /lib64/libc.so.6
      #11 0x0000000000000000 in ?? ()
      (gdb) frame 1
      #1  0x00000000006102d8 in write (this=0x7fffe000bd70, towrite=1260, wattempted=@0x7ffff6c11c78, total_wrote=@0x7ffff6c11c80, 
          buf=<value optimized out>) at ../../iocore/eventsystem/P_UnixSocketManager.h:207
      207	    if (likely((r =::write(fd, buf, size)) >= 0))
      (gdb) print fd
      $3 = 45
      (gdb) print buf
      $4 = (void *) 0x7fffc9860b14
      (gdb) print size
      $5 = <value optimized out>
      (gdb) frame 2
      #2  UnixNetVConnection::load_buffer_and_write (this=0x7fffe000bd70, towrite=1260, wattempted=@0x7ffff6c11c78, 
          total_wrote=@0x7ffff6c11c80, buf=<value optimized out>) at UnixNetVConnection.cc:833
      833	      r = socketManager.write(con.fd, tiovec[0].iov_base, tiovec[0].iov_len);
      (gdb) print con.fd
      $6 = 45
      (gdb) print tiovec[0].iov_base
      $7 = (void *) 0x7fffc9860b14
      (gdb) print tiovec[0].iov_len
      $8 = 1260
      (gdb) frame 1
      #1  0x00000000006102d8 in write (this=0x7fffe000bd70, towrite=1260, wattempted=@0x7ffff6c11c78, total_wrote=@0x7ffff6c11c80, 
          buf=<value optimized out>) at ../../iocore/eventsystem/P_UnixSocketManager.h:207
      207	    if (likely((r =::write(fd, buf, size)) >= 0))
      (gdb) print buf
      $9 = (void *) 0x7fffc9860b14
      (gdb) print *buf
      Attempt to dereference a generic pointer.
      (gdb) frame 2
      #2  UnixNetVConnection::load_buffer_and_write (this=0x7fffe000bd70, towrite=1260, wattempted=@0x7ffff6c11c78, 
          total_wrote=@0x7ffff6c11c80, buf=<value optimized out>) at UnixNetVConnection.cc:833
      833	      r = socketManager.write(con.fd, tiovec[0].iov_base, tiovec[0].iov_len);
      (gdb) print tiovec[0].iov_base
      $10 = (void *) 0x7fffc9860b14
      (gdb) print *((char*)tiovec[0].iov_base)
      $11 = 120 'x'
      (gdb) frame 1
      #1  0x00000000006102d8 in write (this=0x7fffe000bd70, towrite=1260, wattempted=@0x7ffff6c11c78, total_wrote=@0x7ffff6c11c80, 
          buf=<value optimized out>) at ../../iocore/eventsystem/P_UnixSocketManager.h:207
      207	    if (likely((r =::write(fd, buf, size)) >= 0))
      (gdb) print *((char*)buf)
      $12 = 120 'x'
      

        Activity

        Hide
        Leif Hedstrom added a comment -

        Fwiw, I think this crasher only happens with optimized builds (i.e. I can not reproduce with a debug build).

        Show
        Leif Hedstrom added a comment - Fwiw, I think this crasher only happens with optimized builds (i.e. I can not reproduce with a debug build).
        Hide
        Leif Hedstrom added a comment -

        Scratch that last comment, just got a crash with a debug build (just not as easy / fast to reproduce):

        (gdb) bt
        #0  0x0000003f2e60e1fd in write () at ../sysdeps/unix/syscall-template.S:82
        #1  0x0000000000513c63 in SocketManager::write (this=0xe52340, fd=42, buf=0x7fffee0212f7, size=1132, pOLP=0x0)
            at ../iocore/eventsystem/P_UnixSocketManager.h:207
        #2  0x0000000000652cab in UnixNetVConnection::load_buffer_and_write (this=0x7fffc400b820, towrite=1132, 
            wattempted=@0x7fffffffe320, total_wrote=@0x7fffffffe328, buf=...) at UnixNetVConnection.cc:833
        #3  0x0000000000651468 in write_to_net_io (nh=0x7ffff6e0c628, vc=0x7fffc400b820, thread=0x7ffff6e0b010)
            at UnixNetVConnection.cc:439
        #4  0x0000000000650f83 in write_to_net (nh=0x7ffff6e0c628, vc=0x7fffc400b820, pd=0x7ffff611d010, thread=0x7ffff6e0b010)
            at UnixNetVConnection.cc:350
        #5  0x000000000064d8d8 in NetHandler::mainNetEvent (this=0x7ffff6e0c628, event=5, e=0xefa760) at UnixNet.cc:419
        #6  0x00000000004d4ae0 in Continuation::handleEvent (this=0x7ffff6e0c628, event=5, data=0xefa760)
            at ../iocore/eventsystem/I_Continuation.h:146
        #7  0x000000000066e8f8 in EThread::process_event (this=0x7ffff6e0b010, e=0xefa760, calling_code=5) at UnixEThread.cc:140
        #8  0x000000000066eeb5 in EThread::execute (this=0x7ffff6e0b010) at UnixEThread.cc:262
        #9  0x00000000004f9ca0 in main (argc=1, argv=0x7fffffffea68) at Main.cc:1960
        

        Is that "this" pointer really reasonable??

        Show
        Leif Hedstrom added a comment - Scratch that last comment, just got a crash with a debug build (just not as easy / fast to reproduce): (gdb) bt #0 0x0000003f2e60e1fd in write () at ../sysdeps/unix/syscall-template.S:82 #1 0x0000000000513c63 in SocketManager::write ( this =0xe52340, fd=42, buf=0x7fffee0212f7, size=1132, pOLP=0x0) at ../iocore/eventsystem/P_UnixSocketManager.h:207 #2 0x0000000000652cab in UnixNetVConnection::load_buffer_and_write ( this =0x7fffc400b820, towrite=1132, wattempted=@0x7fffffffe320, total_wrote=@0x7fffffffe328, buf=...) at UnixNetVConnection.cc:833 #3 0x0000000000651468 in write_to_net_io (nh=0x7ffff6e0c628, vc=0x7fffc400b820, thread=0x7ffff6e0b010) at UnixNetVConnection.cc:439 #4 0x0000000000650f83 in write_to_net (nh=0x7ffff6e0c628, vc=0x7fffc400b820, pd=0x7ffff611d010, thread=0x7ffff6e0b010) at UnixNetVConnection.cc:350 #5 0x000000000064d8d8 in NetHandler::mainNetEvent ( this =0x7ffff6e0c628, event=5, e=0xefa760) at UnixNet.cc:419 #6 0x00000000004d4ae0 in Continuation::handleEvent ( this =0x7ffff6e0c628, event=5, data=0xefa760) at ../iocore/eventsystem/I_Continuation.h:146 #7 0x000000000066e8f8 in EThread::process_event ( this =0x7ffff6e0b010, e=0xefa760, calling_code=5) at UnixEThread.cc:140 #8 0x000000000066eeb5 in EThread::execute ( this =0x7ffff6e0b010) at UnixEThread.cc:262 #9 0x00000000004f9ca0 in main (argc=1, argv=0x7fffffffea68) at Main.cc:1960 Is that "this" pointer really reasonable??
        Hide
        Leif Hedstrom added a comment -

        Moving these out for v2.1.9 for now.

        Show
        Leif Hedstrom added a comment - Moving these out for v2.1.9 for now.
        Hide
        Leif Hedstrom added a comment -

        Marking this as fixed, after John's changes to the freelist pointers, I'm unable to reproduce this bug. Reopen if this happens again.

        Show
        Leif Hedstrom added a comment - Marking this as fixed, after John's changes to the freelist pointers, I'm unable to reproduce this bug. Reopen if this happens again.

          People

          • Assignee:
            Leif Hedstrom
            Reporter:
            Leif Hedstrom
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development