Bug 30865 - mod_disk_cache leaves many temporary files slowing file system
Summary: mod_disk_cache leaves many temporary files slowing file system
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_cache_disk / mod_disk_cache (show other bugs)
Version: 2.2.24
Hardware: HP Linux
: P3 major with 3 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate, PatchAvailable
: 30866 30867 30868 (view as bug list)
Depends on:
Blocks:
 
Reported: 2004-08-26 14:32 UTC by Patrick Mackinlay
Modified: 2018-11-07 21:09 UTC (History)
2 users (show)



Attachments
Patch against trunk (1.76 KB, patch)
2008-08-12 13:04 UTC, Ruediger Pluem
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Patrick Mackinlay 2004-08-26 14:32:11 UTC
The disk cache module creates temporary files in CacheRoot, these files have the
format aptmp??????
Some of the time it does not clean up and remove the files, so the directory
ends up being full of these files (mostly of 0 size!). Eventually this slows
doen the file system to such a degree that the proxy slows down massively.
Comment 1 Joshua Slive 2004-08-26 15:32:09 UTC
*** Bug 30868 has been marked as a duplicate of this bug. ***
Comment 2 Joshua Slive 2004-08-26 15:32:30 UTC
*** Bug 30867 has been marked as a duplicate of this bug. ***
Comment 3 Joshua Slive 2004-08-26 15:32:45 UTC
*** Bug 30866 has been marked as a duplicate of this bug. ***
Comment 4 Paul Querna 2005-06-03 03:40:11 UTC
Does this still happen with 2.0.54?
Comment 5 RH 2008-04-29 10:42:48 UTC
This still happens in 2.2.8, surprised that it's stagnated this long. This is really a showstopper since it will fill up CacheRoot dir with thousands of these files.
Comment 6 Anton 2008-05-28 09:41:27 UTC
Does nobody use mod_cache?!? This bug is a real bite in the butt one!
Comment 7 Russ Tennant 2008-08-06 22:40:23 UTC
I just noticed this bug on a server. We had 26G of these temporary files.

I compared the error log, access log, and file timestamp and data. The temp files correspond to the following error messages.

(104)Connection reset by peer: cache: error returned while trying to return disk cached data

Running 2.2.4 
Comment 8 rahul 2008-08-12 11:16:40 UTC
can you cat any of the files (of size larger than 0) ?
are they cache headers or data?
Comment 9 Anton 2008-08-12 11:35:22 UTC
I've since gone back to squid, but I can confirm that they are data files.
Comment 10 Russ Tennant 2008-08-12 11:52:35 UTC
They are data files in my case also. I used the data and timestamps to correlate entries in the accesslog by comparing orphaned image data (dimension, visually). I compared the matching accesslog entries with the errorlog to determine that the temp files were orphaned when the error message, "104)Connection reset by peer: cache: error returned while trying to return disk cached data", was logged.

mod_cache # ls aptmp* | head -n 5| xargs file
aptmp03747j: Macromedia Flash data (compressed), version 8
aptmp06aged: Macromedia Flash data (compressed), version 8
aptmp0JnsLu: data
aptmp0TWluZ: Macromedia Flash data (compressed), version 8
aptmp0hE4Mx: Macromedia Flash data (compressed), version 8

mod_cache # ls -l aptmp* | head -n 5
-rw------- 1 apache apache 163680 Aug 12 13:58 aptmp03747j
-rw------- 1 apache apache 139128 Aug 11 03:27 aptmp06aged
-rw------- 1 apache apache   8184 Aug 12 06:04 aptmp0JnsLu
-rw------- 1 apache apache 761112 Aug 11 16:21 aptmp0TWluZ
-rw------- 1 apache apache 965712 Aug 11 17:10 aptmp0hE4Mx
Comment 11 Ruediger Pluem 2008-08-12 13:04:44 UTC
Created attachment 22437 [details]
Patch against trunk

Does the attached patch against trunk (which works against 2.2.9 with some fuzz) resolve your problem?
Comment 12 Graham Leggett 2011-02-27 09:34:55 UTC
This was fixed on httpd-trunk. We now register a pool cleanup, so that we always trigger a cleanup regardless of the type of failure.
Comment 13 issues.apache.org 2013-10-17 07:57:11 UTC
Sorry to disappoint but this is still a problem:

########## Apache -V details ################
Server version: Apache/2.2.24 (Unix)
Server built:   Jun 21 2013 12:00:22
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/u01/apache2.2.24"
 -D SUEXEC_BIN="/u01/apache2.2.24/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"
########## Apache -V details ################

########## Directory Listing ##################
-rw------- 1 apache apache  4943136 Oct 17 08:34 /u01/apache2/cache/aptmpLPAiY8
-rw------- 1 apache apache   139128 Oct 17 08:35 /u01/apache2/cache/aptmpAScO9K
-rw------- 1 apache apache   106392 Oct 17 08:35 /u01/apache2/cache/aptmpzbvoqv
-rw------- 1 apache apache   130944 Oct 17 08:35 /u01/apache2/cache/aptmp4pFbxN
-rw------- 1 apache apache  3846480 Oct 17 08:36 /u01/apache2/cache/aptmpykxaao
-rw------- 1 apache apache   826584 Oct 17 08:38 /u01/apache2/cache/aptmpdgKGKz
-rw------- 1 apache apache 23970936 Oct 17 08:40 /u01/apache2/cache/aptmpxd0Aoy
-rw------- 1 apache apache   695640 Oct 17 08:41 /u01/apache2/cache/aptmpjKoSsz
-rw------- 1 apache apache  7308312 Oct 17 08:41 /u01/apache2/cache/aptmpaY7lOm
-rw------- 1 apache apache  3322704 Oct 17 08:41 /u01/apache2/cache/aptmpXoxnyl
-rw------- 1 apache apache  3224496 Oct 17 08:41 /u01/apache2/cache/aptmpOXqLRq
-rw------- 1 apache apache    98208 Oct 17 08:43 /u01/apache2/cache/aptmpuj8Lnt
-rw------- 1 apache apache   106392 Oct 17 08:43 /u01/apache2/cache/aptmpePy9l1
-rw------- 1 apache apache  3388176 Oct 17 08:45 /u01/apache2/cache/aptmpwdMS1Z
-rw------- 1 apache apache    90024 Oct 17 08:46 /u01/apache2/cache/aptmpeRiaXk
-rw------- 1 apache apache    90024 Oct 17 08:46 /u01/apache2/cache/aptmp0GspIT
-rw------- 1 apache apache    16368 Oct 17 08:47 /u01/apache2/cache/aptmpKZkEoq
-rw------- 1 apache apache    90024 Oct 17 08:47 /u01/apache2/cache/aptmp2QGVm6
-rw------- 1 apache apache   466488 Oct 17 08:49 /u01/apache2/cache/aptmp5TwTBr
-rw------- 1 apache apache 18815016 Oct 17 08:52 /u01/apache2/cache/aptmpYMao0G
-rw------- 1 apache apache  4448256 Oct 17 08:54 /u01/apache2/cache/aptmpW2AscH
-rw------- 1 apache apache  6856704 Oct 17 08:54 /u01/apache2/cache/aptmpVKR2Ac
-rw------- 1 apache apache  3895296 Oct 17 08:54 /u01/apache2/cache/aptmpboQGxr
########## Directory Listing ##################

This is on a pretty busy web server but the number of files grows very quickly and so the htclean I have setup to run a limit cache size to 4GB is effectively useless as things reached 11GB until I started digging into where the "other 7GB" of storage went and discovered these "random" files.

The only possibly interesting thing I'm doing is caching access to images and videos served from Tomcat servers using mod_jk (tomcat-connectors-1.2.37-src.tar.gz to be precise) and running mod_security but I don't see why those would cause the disk cache to leave stray files around.

Again, sorry to disappoint the people who thought they'd fixed this.
Comment 14 issues.apache.org 2013-10-17 08:06:40 UTC
Having just check I can confirm I do not find any of these temporary files on the non production servers which run exactly the same configuration and versions for Apache but experience only internal testing traffic not real users or real load.

Load on the servers is at least 200 requests per second with about a quarter of those going through mod_disk_cache. This is significantly less that the servers can handle however so it's not a server overload causing this.

Also I see not immediate indication that Apache threads or processes are crashing our causing these files to be left lying around (I'm assuming such crashes would appear in the main server error log where it's logs restart).
Comment 15 Eric Covener 2013-10-17 13:31:34 UTC
No indication anyone tried ruedigers patch
Comment 16 Eric Covener 2017-09-08 15:32:25 UTC
I have found one cause of this leaking.  If a response ends abnornally, there
is no original EOS bucket, and ap_die removes the cache_out filter,
so the aptmp file is never cleaned up.
Comment 17 Rainer Jung 2018-02-25 20:36:35 UTC
Undo spam change
Comment 18 William A. Rowe Jr. 2018-11-07 21:09:32 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.