Uploaded image for project: 'Traffic Server'
  1. Traffic Server
  2. TS-1316

ATS connection refused once cache direntries exhausted

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.2.0
    • Fix Version/s: 8.0.0
    • Component/s: Cache
    • Labels:
    • Environment:

      RHEL 6.2 x86_64

      Description

      We had a pair of ATS 3.2.0 boxes that stopped passing traffic simultaneously. Here are the traffic.out msgs we saw on both boxes:

      [Jun 22 05:53:27.637] Server {0x2b6b6d9da700} WARNING: cache directory overflow on '/dev/dm-4' segment 36, purging...
      [Jun 22 05:56:05.542] Server {0x2b6b6d2d3700} WARNING: cache directory overflow on '/dev/dm-4' segment 85, purging...
      [Jun 22 05:56:07.434] Server {0x2b6b6d4d5700} WARNING: cache directory overflow on '/dev/dm-4' segment 71, purging...
      [Jun 22 05:58:24.743] Server {0x2b6b6d8d9700} WARNING: cache directory overflow on '/dev/dm-4' segment 33, purging...
      

      Those messages went on for a couple minutes, then traffic apparently ceased - our monitoring system saw connection refused for port 80 on ATS from then on. The connection refused state went on for many hours until ATS was restarted. There were no traffic_cop msgs in /var/log/messages indicating that the heartbeat failed.

      Here are the relevant ATS settings/stats:

      proxy.process.cache.bytes_total = 190690320384
      proxy.process.cache.direntries.total = 5817752
      proxy.config.cache.min_average_object_size = 32768
      

      We previously came up with proxy.config.cache.min_average_object_size by waiting for the cache to fill and dividing proxy.process.cache.bytes_used by proxy.process.cache.direntries.used - which equals about 34KB.

      We're assuming ATS ran out of direntries and it didn't handle this situation gracefully. As a possible workaround, I'm going to lower proxy.config.cache.min_average_object_size to 24KB.

      Thanks to Bryan Call for helping me troubleshoot this!

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                amc Alan M. Carroll
                Reporter:
                dcarlin David Carlin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: