[TS-968] During/after daily logfile roll, trafficserver seg faults (Sig 11)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 3.0.1
• Fix Version/s: 3.1.4
• Component/s: Logging

Environment

Ubuntu 10.10, 2.6.35-24-virtual

• Assignee: Zhao Yongming
• Reporter: Drew Rothstein
• Created: 25/Sep/11 04:51
• Updated: 06/Apr/12 17:32
• Resolved: 06/Apr/12 17:32

Description

Every day at 00:00:00 after/during the log file roll, I see a segfault. Here are the past couple days:

[Sep 22 00:00:00.000] Server

{47590596851456} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110921.00h00m06s-20110922.00h00m00s.old.
[Sep 22 00:00:00.000] Server {47590596851456}

STATUS: The logfile /usr/local/var/log/trafficserver/squid.log was rolled to /usr/local/var/log/trafficserver/squid.log_trafficserver02.20110921.00h00m01s-20110922.00h00m00s.old.
[Sep 22 00:00:00.000] Server

{47590596851456}

STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110921.00h00m01s-20110922.00h00m00s.old.
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/local/bin/traffic_server - STACK TRACE:
[Sep 22 00:00:17.729] Manager

{140722071643936} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Sep 22 00:00:17.729] Manager {140722071643936}

FATAL: (last system error 104: Connection reset by peer)
[Sep 22 00:00:17.730] Manager

{140722071643936} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
[Sep 22 00:00:17.730] Manager {140722071643936}

NOTE: [LocalManager::processShutdown] Executing process shutdown request.
[Sep 22 00:00:17.730] Manager

{140722071643936} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep 22 00:00:17.730] Manager {140722071643936}

ERROR: (last system error 32: Broken pipe)
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
[Sep 22 00:00:17.786]

{140131209512736} STATUS: opened /usr/local/var/log/trafficserver/manager.log
[Sep 22 00:00:17.786] {140131209512736}

NOTE: updated diags config
[Sep 22 00:00:17.805] Manager

{140131209512736} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.35-24-virtual'
[Sep 22 00:00:17.805] Manager {140131209512736}

NOTE: [LocalManager::listenForProxy] Listening on port: 80
[Sep 22 00:00:17.805] Manager

{140131209512736} NOTE: [TrafficManager] Setup complete
[Sep 22 00:00:18.827] Manager {140131209512736}

NOTE: [LocalManager::startProxy] Launching ts process
[TrafficServer] using root directory '/usr/local'
[Sep 22 00:00:18.849] Manager

{140131209512736} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
[Sep 22 00:00:18.849] Manager {140131209512736}

NOTE: [Alarms::signalAlarm] Server Process born
[Sep 22 00:00:19.874]

{47510015031936} STATUS: opened /usr/local/var/log/trafficserver/diags.log
[Sep 22 00:00:19.874] {47510015031936}

NOTE: updated diags config
[Sep 22 00:00:19.879] Server

{47510015031936} NOTE: cache clustering disabled
[Sep 22 00:00:19.908] Server {47510015031936}

NOTE: cache clustering disabled
[Sep 22 00:00:20.019] Server

{47510015031936} NOTE: logging initialized[7], logging_mode = 3
[Sep 22 00:00:20.032] Server {47510015031936}

NOTE: traffic server running
[Sep 22 00:00:20.045] Server

{47510028859136}

NOTE: cache enabled

[Sep 23 00:00:00.000] Server

{47409990321920} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110922.00h00m11s-20110923.00h00m00s.old.
[Sep 23 00:00:00.000] Server {47409990321920}

STATUS: The logfile /usr/local/var/log/trafficserver/squid.log was rolled to /usr/local/var/log/trafficserver/squid.log_trafficserver02.20110922.00h00m06s-20110923.00h00m00s.old.
[Sep 23 00:00:00.000] Server

{47409990321920}

STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110922.00h00m06s-20110923.00h00m00s.old.
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/local/bin/traffic_server - STACK TRACE:
[Sep 23 00:00:14.668] Manager

{140131209512736} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Sep 23 00:00:14.668] Manager {140131209512736}

FATAL: (last system error 104: Connection reset by peer)
[Sep 23 00:00:14.668] Manager

{140131209512736} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
[Sep 23 00:00:14.668] Manager {140131209512736}

NOTE: [LocalManager::processShutdown] Executing process shutdown request.
[Sep 23 00:00:14.668] Manager

{140131209512736} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep 23 00:00:14.668] Manager {140131209512736}

ERROR: (last system error 32: Broken pipe)
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
[Sep 23 00:00:14.701]

{140532573161248} STATUS: opened /usr/local/var/log/trafficserver/manager.log
[Sep 23 00:00:14.701] {140532573161248}

NOTE: updated diags config
[Sep 23 00:00:14.703] Manager

{140532573161248} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.35-24-virtual'
[Sep 23 00:00:14.703] Manager {140532573161248}

NOTE: [LocalManager::listenForProxy] Listening on port: 80
[Sep 23 00:00:14.703] Manager

{140532573161248} NOTE: [TrafficManager] Setup complete
[Sep 23 00:00:15.734] Manager {140532573161248}

NOTE: [LocalManager::startProxy] Launching ts process
[TrafficServer] using root directory '/usr/local'
[Sep 23 00:00:15.757] Manager

{140532573161248} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
[Sep 23 00:00:15.757] Manager {140532573161248}

NOTE: [Alarms::signalAlarm] Server Process born
[Sep 23 00:00:16.782]

{47629954202240} STATUS: opened /usr/local/var/log/trafficserver/diags.log
[Sep 23 00:00:16.783] {47629954202240}

NOTE: updated diags config
[Sep 23 00:00:16.788] Server

{47629954202240} NOTE: cache clustering disabled
[Sep 23 00:00:16.817] Server {47629954202240}

NOTE: cache clustering disabled
[Sep 23 00:00:16.931] Server

{47629954202240} NOTE: logging initialized[7], logging_mode = 3
[Sep 23 00:00:16.942] Server {47629954202240}

NOTE: traffic server running
[Sep 23 00:00:17.078] Server

{47629969082112}

NOTE: cache enabled

[Sep 24 00:00:00.000] Server

{47805662865152} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110923.00h00m06s-20110924.00h00m00s.old.
[Sep 24 00:00:00.000] Server {47805662865152}

STATUS: The logfile /usr/local/var/log/trafficserver/squid.blog was rolled to /usr/local/var/log/trafficserver/squid.blog_trafficserver02.20110923.20h41m21s-20110924.00h00m00s.old.
[Sep 24 00:00:00.000] Server

{47805662865152}

STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110923.00h00m02s-20110924.00h00m00s.old.
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/local/bin/traffic_server - STACK TRACE:
[Sep 24 00:00:19.970] Manager

{140532573161248} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
[Sep 24 00:00:19.970] Manager {140532573161248}

ERROR: (last system error 2: No such file or directory)
[Sep 24 00:00:19.970] Manager

{140532573161248} ERROR: [Alarms::signalAlarm] Server Process was reset
[Sep 24 00:00:19.970] Manager {140532573161248}

ERROR: (last system error 2: No such file or directory)
[Sep 24 00:00:20.976] Manager

{140532573161248} NOTE: [LocalManager::startProxy] Launching ts process
[TrafficServer] using root directory '/usr/local'
[Sep 24 00:00:21.004] Manager {140532573161248}

NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '9'
[Sep 24 00:00:21.005] Manager

{140532573161248}

NOTE: [Alarms::signalAlarm] Server Process born
[Sep 24 00:00:22.018]

{47227784231552} STATUS: opened /usr/local/var/log/trafficserver/diags.log
[Sep 24 00:00:22.018] {47227784231552}

NOTE: updated diags config
[Sep 24 00:00:22.023] Server

{47227784231552} NOTE: cache clustering disabled
[Sep 24 00:00:22.050] Server {47227784231552}

NOTE: cache clustering disabled
[Sep 24 00:00:22.164] Server

{47227784231552} NOTE: logging initialized[7], logging_mode = 3
[Sep 24 00:00:22.177] Server {47227784231552}

NOTE: traffic server running
[Sep 24 00:00:22.381] Server

{47227799111424}

NOTE: cache enabled

[Sep 25 00:00:00.000] Server

{47877604226816} STATUS: The logfile /usr/local/var/log/trafficserver/error.log was rolled to /usr/local/var/log/trafficserver/error.log_trafficserver02.20110924.00h00m11s-20110925.00h00m00s.old.
[Sep 25 00:00:00.000] Server {47877604226816}

STATUS: The logfile /usr/local/var/log/trafficserver/squid.blog was rolled to /usr/local/var/log/trafficserver/squid.blog_trafficserver02.20110924.00h00m06s-20110925.00h00m00s.old.
[Sep 25 00:00:00.000] Server

{47877604226816}

STATUS: The logfile /usr/local/var/log/trafficserver/extended2.log was rolled to /usr/local/var/log/trafficserver/extended2.log_trafficserver02.20110924.00h00m02s-20110925.00h00m00s.old.
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/local/bin/traffic_server - STACK TRACE:
[Sep 25 00:00:19.597] Manager

{139721826051872} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
[Sep 25 00:00:19.597] Manager {139721826051872}

ERROR: (last system error 2: No such file or directory)
[Sep 25 00:00:19.597] Manager

{139721826051872} ERROR: [Alarms::signalAlarm] Server Process was reset
[Sep 25 00:00:19.597] Manager {139721826051872}

ERROR: (last system error 2: No such file or directory)
[Sep 25 00:00:20.605] Manager

{139721826051872} NOTE: [LocalManager::startProxy] Launching ts process
[TrafficServer] using root directory '/usr/local'
[Sep 25 00:00:20.627] Manager {139721826051872}

NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '11'
[Sep 25 00:00:20.627] Manager

{139721826051872}

NOTE: [Alarms::signalAlarm] Server Process born
[Sep 25 00:00:21.641]

{47430015421056} STATUS: opened /usr/local/var/log/trafficserver/diags.log
[Sep 25 00:00:21.641] {47430015421056}

NOTE: updated diags config
[Sep 25 00:00:21.647] Server

{47430015421056} NOTE: cache clustering disabled
[Sep 25 00:00:21.673] Server {47430015421056}

NOTE: cache clustering disabled
[Sep 25 00:00:21.787] Server

{47430015421056} NOTE: logging initialized[7], logging_mode = 3
[Sep 25 00:00:21.799] Server {47430015421056}

NOTE: traffic server running
[Sep 25 00:00:22.020] Server

{47430030300928}

NOTE: cache enabled