Traffic Server
  1. Traffic Server
  2. TS-968

During/after daily logfile roll, trafficserver seg faults (Sig 11)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.1.4
    • Component/s: Logging
    • Labels:
      None
    • Environment:

      Ubuntu 10.10, 2.6.35-24-virtual

      Description

      Every day at 00:00:00 after/during the log file roll, I see a segfault. Here are the past couple days:

      [Sep 22 00:00:00.000] Server

      {47590596851456} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110921.00h00m06s-20110922.00h00m00s.old.
      [Sep 22 00:00:00.000] Server {47590596851456}

      STATUS: The logfile /usr/local/var/log/trafficserver/squid.log was rolled to /usr/local/var/log/trafficserver/squid.log_trafficserver02.20110921.00h00m01s-20110922.00h00m00s.old.
      [Sep 22 00:00:00.000] Server

      {47590596851456}

      STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110921.00h00m01s-20110922.00h00m00s.old.
      NOTE: Traffic Server received Sig 11: Segmentation fault
      /usr/local/bin/traffic_server - STACK TRACE:
      [Sep 22 00:00:17.729] Manager

      {140722071643936} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
      [Sep 22 00:00:17.729] Manager {140722071643936}

      FATAL: (last system error 104: Connection reset by peer)
      [Sep 22 00:00:17.730] Manager

      {140722071643936} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
      [Sep 22 00:00:17.730] Manager {140722071643936}

      NOTE: [LocalManager::processShutdown] Executing process shutdown request.
      [Sep 22 00:00:17.730] Manager

      {140722071643936} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
      [Sep 22 00:00:17.730] Manager {140722071643936}

      ERROR: (last system error 32: Broken pipe)
      [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
      [Sep 22 00:00:17.786]

      {140131209512736} STATUS: opened /usr/local/var/log/trafficserver/manager.log
      [Sep 22 00:00:17.786] {140131209512736}

      NOTE: updated diags config
      [Sep 22 00:00:17.805] Manager

      {140131209512736} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.35-24-virtual'
      [Sep 22 00:00:17.805] Manager {140131209512736}

      NOTE: [LocalManager::listenForProxy] Listening on port: 80
      [Sep 22 00:00:17.805] Manager

      {140131209512736} NOTE: [TrafficManager] Setup complete
      [Sep 22 00:00:18.827] Manager {140131209512736}

      NOTE: [LocalManager::startProxy] Launching ts process
      [TrafficServer] using root directory '/usr/local'
      [Sep 22 00:00:18.849] Manager

      {140131209512736} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
      [Sep 22 00:00:18.849] Manager {140131209512736}

      NOTE: [Alarms::signalAlarm] Server Process born
      [Sep 22 00:00:19.874]

      {47510015031936} STATUS: opened /usr/local/var/log/trafficserver/diags.log
      [Sep 22 00:00:19.874] {47510015031936}

      NOTE: updated diags config
      [Sep 22 00:00:19.879] Server

      {47510015031936} NOTE: cache clustering disabled
      [Sep 22 00:00:19.908] Server {47510015031936}

      NOTE: cache clustering disabled
      [Sep 22 00:00:20.019] Server

      {47510015031936} NOTE: logging initialized[7], logging_mode = 3
      [Sep 22 00:00:20.032] Server {47510015031936}

      NOTE: traffic server running
      [Sep 22 00:00:20.045] Server

      {47510028859136}

      NOTE: cache enabled

      [Sep 23 00:00:00.000] Server

      {47409990321920} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110922.00h00m11s-20110923.00h00m00s.old.
      [Sep 23 00:00:00.000] Server {47409990321920}

      STATUS: The logfile /usr/local/var/log/trafficserver/squid.log was rolled to /usr/local/var/log/trafficserver/squid.log_trafficserver02.20110922.00h00m06s-20110923.00h00m00s.old.
      [Sep 23 00:00:00.000] Server

      {47409990321920}

      STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110922.00h00m06s-20110923.00h00m00s.old.
      NOTE: Traffic Server received Sig 11: Segmentation fault
      /usr/local/bin/traffic_server - STACK TRACE:
      [Sep 23 00:00:14.668] Manager

      {140131209512736} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
      [Sep 23 00:00:14.668] Manager {140131209512736}

      FATAL: (last system error 104: Connection reset by peer)
      [Sep 23 00:00:14.668] Manager

      {140131209512736} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
      [Sep 23 00:00:14.668] Manager {140131209512736}

      NOTE: [LocalManager::processShutdown] Executing process shutdown request.
      [Sep 23 00:00:14.668] Manager

      {140131209512736} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
      [Sep 23 00:00:14.668] Manager {140131209512736}

      ERROR: (last system error 32: Broken pipe)
      [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
      [Sep 23 00:00:14.701]

      {140532573161248} STATUS: opened /usr/local/var/log/trafficserver/manager.log
      [Sep 23 00:00:14.701] {140532573161248}

      NOTE: updated diags config
      [Sep 23 00:00:14.703] Manager

      {140532573161248} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.35-24-virtual'
      [Sep 23 00:00:14.703] Manager {140532573161248}

      NOTE: [LocalManager::listenForProxy] Listening on port: 80
      [Sep 23 00:00:14.703] Manager

      {140532573161248} NOTE: [TrafficManager] Setup complete
      [Sep 23 00:00:15.734] Manager {140532573161248}

      NOTE: [LocalManager::startProxy] Launching ts process
      [TrafficServer] using root directory '/usr/local'
      [Sep 23 00:00:15.757] Manager

      {140532573161248} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
      [Sep 23 00:00:15.757] Manager {140532573161248}

      NOTE: [Alarms::signalAlarm] Server Process born
      [Sep 23 00:00:16.782]

      {47629954202240} STATUS: opened /usr/local/var/log/trafficserver/diags.log
      [Sep 23 00:00:16.783] {47629954202240}

      NOTE: updated diags config
      [Sep 23 00:00:16.788] Server

      {47629954202240} NOTE: cache clustering disabled
      [Sep 23 00:00:16.817] Server {47629954202240}

      NOTE: cache clustering disabled
      [Sep 23 00:00:16.931] Server

      {47629954202240} NOTE: logging initialized[7], logging_mode = 3
      [Sep 23 00:00:16.942] Server {47629954202240}

      NOTE: traffic server running
      [Sep 23 00:00:17.078] Server

      {47629969082112}

      NOTE: cache enabled

      [Sep 24 00:00:00.000] Server

      {47805662865152} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110923.00h00m06s-20110924.00h00m00s.old.
      [Sep 24 00:00:00.000] Server {47805662865152}

      STATUS: The logfile /usr/local/var/log/trafficserver/squid.blog was rolled to /usr/local/var/log/trafficserver/squid.blog_trafficserver02.20110923.20h41m21s-20110924.00h00m00s.old.
      [Sep 24 00:00:00.000] Server

      {47805662865152}

      STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110923.00h00m02s-20110924.00h00m00s.old.
      NOTE: Traffic Server received Sig 11: Segmentation fault
      /usr/local/bin/traffic_server - STACK TRACE:
      [Sep 24 00:00:19.970] Manager

      {140532573161248} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
      [Sep 24 00:00:19.970] Manager {140532573161248}

      ERROR: (last system error 2: No such file or directory)
      [Sep 24 00:00:19.970] Manager

      {140532573161248} ERROR: [Alarms::signalAlarm] Server Process was reset
      [Sep 24 00:00:19.970] Manager {140532573161248}

      ERROR: (last system error 2: No such file or directory)
      [Sep 24 00:00:20.976] Manager

      {140532573161248} NOTE: [LocalManager::startProxy] Launching ts process
      [TrafficServer] using root directory '/usr/local'
      [Sep 24 00:00:21.004] Manager {140532573161248}

      NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '9'
      [Sep 24 00:00:21.005] Manager

      {140532573161248}

      NOTE: [Alarms::signalAlarm] Server Process born
      [Sep 24 00:00:22.018]

      {47227784231552} STATUS: opened /usr/local/var/log/trafficserver/diags.log
      [Sep 24 00:00:22.018] {47227784231552}

      NOTE: updated diags config
      [Sep 24 00:00:22.023] Server

      {47227784231552} NOTE: cache clustering disabled
      [Sep 24 00:00:22.050] Server {47227784231552}

      NOTE: cache clustering disabled
      [Sep 24 00:00:22.164] Server

      {47227784231552} NOTE: logging initialized[7], logging_mode = 3
      [Sep 24 00:00:22.177] Server {47227784231552}

      NOTE: traffic server running
      [Sep 24 00:00:22.381] Server

      {47227799111424}

      NOTE: cache enabled

      [Sep 25 00:00:00.000] Server

      {47877604226816} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110924.00h00m11s-20110925.00h00m00s.old.
      [Sep 25 00:00:00.000] Server {47877604226816}

      STATUS: The logfile /usr/local/var/log/trafficserver/squid.blog was rolled to /usr/local/var/log/trafficserver/squid.blog_trafficserver02.20110924.00h00m06s-20110925.00h00m00s.old.
      [Sep 25 00:00:00.000] Server

      {47877604226816}

      STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110924.00h00m02s-20110925.00h00m00s.old.
      NOTE: Traffic Server received Sig 11: Segmentation fault
      /usr/local/bin/traffic_server - STACK TRACE:
      [Sep 25 00:00:19.597] Manager

      {139721826051872} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
      [Sep 25 00:00:19.597] Manager {139721826051872}

      ERROR: (last system error 2: No such file or directory)
      [Sep 25 00:00:19.597] Manager

      {139721826051872} ERROR: [Alarms::signalAlarm] Server Process was reset
      [Sep 25 00:00:19.597] Manager {139721826051872}

      ERROR: (last system error 2: No such file or directory)
      [Sep 25 00:00:20.605] Manager

      {139721826051872} NOTE: [LocalManager::startProxy] Launching ts process
      [TrafficServer] using root directory '/usr/local'
      [Sep 25 00:00:20.627] Manager {139721826051872}

      NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '11'
      [Sep 25 00:00:20.627] Manager

      {139721826051872}

      NOTE: [Alarms::signalAlarm] Server Process born
      [Sep 25 00:00:21.641]

      {47430015421056} STATUS: opened /usr/local/var/log/trafficserver/diags.log
      [Sep 25 00:00:21.641] {47430015421056}

      NOTE: updated diags config
      [Sep 25 00:00:21.647] Server

      {47430015421056} NOTE: cache clustering disabled
      [Sep 25 00:00:21.673] Server {47430015421056}

      NOTE: cache clustering disabled
      [Sep 25 00:00:21.787] Server

      {47430015421056} NOTE: logging initialized[7], logging_mode = 3
      [Sep 25 00:00:21.799] Server {47430015421056}

      NOTE: traffic server running
      [Sep 25 00:00:22.020] Server

      {47430030300928}

      NOTE: cache enabled

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Zhao Yongming
            Reporter:
            Drew Rothstein
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development