Uploaded image for project: 'Traffic Server'
  1. Traffic Server
  2. TS-1272

workaround for - FATAL: HttpSM.cc:890: failed assert `0`

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.1.3
    • 3.1.4
    • HTTP
    • Linux hostname.domain.com 2.6.32-220.el6.x86_64 #1 SMP Wed Dec 7 10:41:06 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

    Description

      We have a 3.1.3 installation running against a light production load and are occasionally getting this failed assertion:

      FATAL: HttpSM.cc:890: failed assert `0`
      /usr/bin/traffic_server - STACK TRACE:
      /usr/lib64/trafficserver/libtsutil.so.3(ink_fatal+0x88)[0x2b3b69c88538]
      /usr/lib64/trafficserver/libtsutil.so.3(_ink_assert+0x1f)[0x2b3b69c86c6f]
      /usr/bin/traffic_server(_ZN6HttpSM28state_watch_for_client_abortEiPv+0x7d)[0x52ae0d]
      /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8)[0x536008]
      /usr/bin/traffic_server(_ZN18UnixNetVConnection14readSignalDoneEiP10NetHandler+0x3d)[0x678e2d]
      /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x6ef)[0x66cabf]
      /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x25a)[0x6737ba]
      /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0xb4)[0x69cc34]
      /usr/bin/traffic_server(_ZN7EThread7executeEv+0x4c3)[0x69d5c3]
      /usr/bin/traffic_server[0x69bb72]
      /lib64/libpthread.so.0[0x3170e077f1]
      /lib64/libc.so.6(clone+0x6d)[0x3170ae570d]

      This causes a corruption of the disk cache every time it occurs, such that whenever the traffic server starts back up, it gets into a crash loop and does this over and over each time it tries to start up:

      NOTE: Traffic Server received Sig 11: Segmentation fault
      /usr/bin/traffic_server - STACK TRACE:
      /lib64/libpthread.so.0[0x3170e0f4a0]
      /usr/bin/traffic_server(_Z15dir_clear_rangellP3Vol+0x6b)[0x63370b]
      /usr/bin/traffic_server(_ZN3Vol24handle_recover_from_dataEiPv+0x531)[0x62c5d1]
      /usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x35)[0x62cfb5]
      /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0xb4)[0x69cc34]
      /usr/bin/traffic_server(_ZN7EThread7executeEv+0x5ab)[0x69d6ab]
      /usr/bin/traffic_server[0x69bb72]
      /lib64/libpthread.so.0[0x3170e077f1]
      /lib64/libc.so.6(clone+0x6d)[0x3170ae570d]
      [Apr 29 17:39:57.149] Manager

      {0x7f4d3481c7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
      [Apr 29 17:39:57.149] Manager {0x7f4d3481c7e0}

      FATAL: (last system error 104: Connection reset by peer)
      [Apr 29 17:39:57.149] Manager

      {0x7f4d3481c7e0} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
      [Apr 29 17:39:57.149] Manager {0x7f4d3481c7e0}

      NOTE: [LocalManager::processShutdown] Executing process shutdown request.
      [Apr 29 17:39:57.149] Manager

      {0x7f4d3481c7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
      [Apr 29 17:39:57.149] Manager {0x7f4d3481c7e0}

      ERROR: (last system error 32: Broken pipe)

      The configuration for this setup is pretty minimal, and is mostly a single remap rule that chains together the remap_regex and conf_remap plugins to turn off auth for files ending in a particular extension.

      I will add any appropriate configs, logs, etc as needed to help troubleshoot this.

      Attachments

        Issue Links

          Activity

            People

              jamespeach James Peach
              jamespeach James Peach
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: