Traffic Server
  1. Traffic Server
  2. TS-981

Remove libev support (it's not well support, and might crash)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.1.4
    • Component/s: Core
    • Labels:
      None
    • Environment:

      4 core desktop. Centos 5.6, kernel 2.6.39.2 compiled with TPROXY support.
      ATS compiled as: ./configure --enable-tproxy --enable-libev

      Description

      ATS crashes under heavy load testing - around 25,000 HTTP transactions per second.
      I have tested it on both vesions 3.0.0 and 3.0.1 and the same happens.

      GDB info:

      Program received signal SIGSEGV, Segmentation fault.
      [Switching to Thread 0x42174940 (LWP 6663)]
      0x000000000068fd4b in modify (this=0x2aaaab12c628, event=<value optimized out>, e=<value optimized out>) at P_UnixNet.h:536
      536 fd_change(event_loop->eio, eio.fd, 0);
      (gdb) bt
      #0 0x000000000068fd4b in modify (this=0x2aaaab12c628, event=<value optimized out>, e=<value optimized out>) at P_UnixNet.h:536
      #1 NetHandler::mainNetEvent (this=0x2aaaab12c628, event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:432
      #2 0x00000000006bfc4f in EThread::process_event (this=0x2aaaab12b010, e=0xf44570, calling_code=5) at I_Continuation.h:146
      #3 0x00000000006c055c in EThread::execute (this=0x2aaaab12b010) at UnixEThread.cc:262
      #4 0x00000000006bef8e in spawn_thread_internal (a=0xf35c90) at Thread.cc:88
      #5 0x0000003e9dc0673d in start_thread () from /lib64/libpthread.so.0
      #6 0x0000003e9d0d44bd in clone () from /lib64/libc.so.6
      (gdb) info f
      Stack level 0, frame at 0x42174030:
      rip = 0x68fd4b in modify (P_UnixNet.h:536); saved rip 0x6bfc4f
      inlined into frame 1
      source language c++.
      Arglist at unknown address.
      Locals at unknown address, Previous frame's sp in rsp

      1. records.config
        27 kB
        Danny Shporer

        Activity

        Danny Shporer created issue -
        Hide
        Danny Shporer added a comment -

        Config file used when ATS crashed

        Show
        Danny Shporer added a comment - Config file used when ATS crashed
        Danny Shporer made changes -
        Field Original Value New Value
        Attachment records.config [ 12498856 ]
        Leif Hedstrom made changes -
        Fix Version/s 3.1.2 [ 12317605 ]
        Hide
        Leif Hedstrom added a comment -

        I'm curious if you have only got this to reproduce in transparent proxy, or if it happens in reverse or forward proxy as well? Also, can you reproduce it with a debug build? It would (hopefully) have more information.

        Show
        Leif Hedstrom added a comment - I'm curious if you have only got this to reproduce in transparent proxy, or if it happens in reverse or forward proxy as well? Also, can you reproduce it with a debug build? It would (hopefully) have more information.
        Hide
        Danny Shporer added a comment -

        I have not tried to reproduce with reverse or forward proxy.
        I did try to reproduce in debug build but without success - my tool was not able to reach 25K TPS in debug build. It only reached around 16K TPS and there was no exception.

        The problem seems to be with the libev support in ATS. Without it in my Linux environment when plain epoll is used (./configure --enable-tproxy) it does not happen and I have been working this way since (I have reached round 34K TPS).
        There is another issue regarding heavy load in transparent proxy - the Linux TPROXY stack only takes into account the local addresses when using dynamic bind (bind without specifying a port). I opened a new bug for this one - TS-1075.

        Show
        Danny Shporer added a comment - I have not tried to reproduce with reverse or forward proxy. I did try to reproduce in debug build but without success - my tool was not able to reach 25K TPS in debug build. It only reached around 16K TPS and there was no exception. The problem seems to be with the libev support in ATS. Without it in my Linux environment when plain epoll is used (./configure --enable-tproxy) it does not happen and I have been working this way since (I have reached round 34K TPS). There is another issue regarding heavy load in transparent proxy - the Linux TPROXY stack only takes into account the local addresses when using dynamic bind (bind without specifying a port). I opened a new bug for this one - TS-1075 .
        Hide
        Leif Hedstrom added a comment -

        Not sure I understand, are you saying you are using the libev "support"? I thought we removed that, because it was not working well, and not as fast as epoll.

        Show
        Leif Hedstrom added a comment - Not sure I understand, are you saying you are using the libev "support"? I thought we removed that, because it was not working well, and not as fast as epoll.
        Hide
        Danny Shporer added a comment -

        Yes I was. It's mentioned in the Environment field of the bug report details that ATS is compiled with --enable-libev.
        I did not find anywhere any mention that it has been removed.
        Anyway, when I realized that there was an exception in the libev code, I reverted to using epoll and that seems to work penfing bug report TS-1075.
        If libev is not supported by ATS then I guess you can close this bug. I would suggest to make it clear not to use it (maybe remove README.libev from the code distributable).

        Show
        Danny Shporer added a comment - Yes I was. It's mentioned in the Environment field of the bug report details that ATS is compiled with --enable-libev. I did not find anywhere any mention that it has been removed. Anyway, when I realized that there was an exception in the libev code, I reverted to using epoll and that seems to work penfing bug report TS-1075 . If libev is not supported by ATS then I guess you can close this bug. I would suggest to make it clear not to use it (maybe remove README.libev from the code distributable).
        Hide
        Leif Hedstrom added a comment -

        Ah, yeah, I see it's still there :/. Let me take a look at it, and see what is there. I'm fairly certain the person who did the libev code decided to give up, and that it was fairly pointless.

        Show
        Leif Hedstrom added a comment - Ah, yeah, I see it's still there :/. Let me take a look at it, and see what is there. I'm fairly certain the person who did the libev code decided to give up, and that it was fairly pointless.
        Leif Hedstrom made changes -
        Fix Version/s 3.1.3 [ 12317969 ]
        Fix Version/s 3.1.2 [ 12317605 ]
        Hide
        Leif Hedstrom added a comment -

        Moved to 3.1.4, please move bugs back to 3.1.3, which you will work on in the next 2 weeks.

        Show
        Leif Hedstrom added a comment - Moved to 3.1.4, please move bugs back to 3.1.3, which you will work on in the next 2 weeks.
        Leif Hedstrom made changes -
        Fix Version/s 3.1.4 [ 12318543 ]
        Fix Version/s 3.1.3 [ 12317969 ]
        Leif Hedstrom made changes -
        Fix Version/s 3.1.4 [ 12318543 ]
        Fix Version/s 3.1.5 [ 12320056 ]
        Leif Hedstrom made changes -
        Fix Version/s 3.1.4 [ 12318543 ]
        Fix Version/s 3.1.5 [ 12320056 ]
        Leif Hedstrom made changes -
        Assignee Leif Hedstrom [ zwoop ]
        Hide
        Leif Hedstrom added a comment -

        I'll check with John, but I think we should just remove libev entirely from the source.

        Show
        Leif Hedstrom added a comment - I'll check with John, but I think we should just remove libev entirely from the source.
        Hide
        Leif Hedstrom added a comment -

        Changing the summary on this one, as we'll remove libev support (I checked with John on this).

        Show
        Leif Hedstrom added a comment - Changing the summary on this one, as we'll remove libev support (I checked with John on this).
        Leif Hedstrom made changes -
        Summary ATS as transparent proxy crashes under heavy load Remove libev support (it's not well support, and might crash)
        Leif Hedstrom made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Leif Hedstrom made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Leif Hedstrom
            Reporter:
            Danny Shporer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development