The disk cache module creates temporary files in CacheRoot, these files have the format aptmp?????? Some of the time it does not clean up and remove the files, so the directory ends up being full of these files (mostly of 0 size!). Eventually this slows doen the file system to such a degree that the proxy slows down massively.
*** Bug 30868 has been marked as a duplicate of this bug. ***
*** Bug 30867 has been marked as a duplicate of this bug. ***
*** Bug 30866 has been marked as a duplicate of this bug. ***
Does this still happen with 2.0.54?
This still happens in 2.2.8, surprised that it's stagnated this long. This is really a showstopper since it will fill up CacheRoot dir with thousands of these files.
Does nobody use mod_cache?!? This bug is a real bite in the butt one!
I just noticed this bug on a server. We had 26G of these temporary files. I compared the error log, access log, and file timestamp and data. The temp files correspond to the following error messages. (104)Connection reset by peer: cache: error returned while trying to return disk cached data Running 2.2.4
can you cat any of the files (of size larger than 0) ? are they cache headers or data?
I've since gone back to squid, but I can confirm that they are data files.
They are data files in my case also. I used the data and timestamps to correlate entries in the accesslog by comparing orphaned image data (dimension, visually). I compared the matching accesslog entries with the errorlog to determine that the temp files were orphaned when the error message, "104)Connection reset by peer: cache: error returned while trying to return disk cached data", was logged. mod_cache # ls aptmp* | head -n 5| xargs file aptmp03747j: Macromedia Flash data (compressed), version 8 aptmp06aged: Macromedia Flash data (compressed), version 8 aptmp0JnsLu: data aptmp0TWluZ: Macromedia Flash data (compressed), version 8 aptmp0hE4Mx: Macromedia Flash data (compressed), version 8 mod_cache # ls -l aptmp* | head -n 5 -rw------- 1 apache apache 163680 Aug 12 13:58 aptmp03747j -rw------- 1 apache apache 139128 Aug 11 03:27 aptmp06aged -rw------- 1 apache apache 8184 Aug 12 06:04 aptmp0JnsLu -rw------- 1 apache apache 761112 Aug 11 16:21 aptmp0TWluZ -rw------- 1 apache apache 965712 Aug 11 17:10 aptmp0hE4Mx
Created attachment 22437 [details] Patch against trunk Does the attached patch against trunk (which works against 2.2.9 with some fuzz) resolve your problem?
This was fixed on httpd-trunk. We now register a pool cleanup, so that we always trigger a cleanup regardless of the type of failure.
Sorry to disappoint but this is still a problem: ########## Apache -V details ################ Server version: Apache/2.2.24 (Unix) Server built: Jun 21 2013 12:00:22 Server's Module Magic Number: 20051115:31 Server loaded: APR 1.2.7, APR-Util 1.2.7 Compiled using: APR 1.2.7, APR-Util 1.2.7 Architecture: 64-bit Server MPM: Worker threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/worker" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/u01/apache2.2.24" -D SUEXEC_BIN="/u01/apache2.2.24/bin/suexec" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf" ########## Apache -V details ################ ########## Directory Listing ################## -rw------- 1 apache apache 4943136 Oct 17 08:34 /u01/apache2/cache/aptmpLPAiY8 -rw------- 1 apache apache 139128 Oct 17 08:35 /u01/apache2/cache/aptmpAScO9K -rw------- 1 apache apache 106392 Oct 17 08:35 /u01/apache2/cache/aptmpzbvoqv -rw------- 1 apache apache 130944 Oct 17 08:35 /u01/apache2/cache/aptmp4pFbxN -rw------- 1 apache apache 3846480 Oct 17 08:36 /u01/apache2/cache/aptmpykxaao -rw------- 1 apache apache 826584 Oct 17 08:38 /u01/apache2/cache/aptmpdgKGKz -rw------- 1 apache apache 23970936 Oct 17 08:40 /u01/apache2/cache/aptmpxd0Aoy -rw------- 1 apache apache 695640 Oct 17 08:41 /u01/apache2/cache/aptmpjKoSsz -rw------- 1 apache apache 7308312 Oct 17 08:41 /u01/apache2/cache/aptmpaY7lOm -rw------- 1 apache apache 3322704 Oct 17 08:41 /u01/apache2/cache/aptmpXoxnyl -rw------- 1 apache apache 3224496 Oct 17 08:41 /u01/apache2/cache/aptmpOXqLRq -rw------- 1 apache apache 98208 Oct 17 08:43 /u01/apache2/cache/aptmpuj8Lnt -rw------- 1 apache apache 106392 Oct 17 08:43 /u01/apache2/cache/aptmpePy9l1 -rw------- 1 apache apache 3388176 Oct 17 08:45 /u01/apache2/cache/aptmpwdMS1Z -rw------- 1 apache apache 90024 Oct 17 08:46 /u01/apache2/cache/aptmpeRiaXk -rw------- 1 apache apache 90024 Oct 17 08:46 /u01/apache2/cache/aptmp0GspIT -rw------- 1 apache apache 16368 Oct 17 08:47 /u01/apache2/cache/aptmpKZkEoq -rw------- 1 apache apache 90024 Oct 17 08:47 /u01/apache2/cache/aptmp2QGVm6 -rw------- 1 apache apache 466488 Oct 17 08:49 /u01/apache2/cache/aptmp5TwTBr -rw------- 1 apache apache 18815016 Oct 17 08:52 /u01/apache2/cache/aptmpYMao0G -rw------- 1 apache apache 4448256 Oct 17 08:54 /u01/apache2/cache/aptmpW2AscH -rw------- 1 apache apache 6856704 Oct 17 08:54 /u01/apache2/cache/aptmpVKR2Ac -rw------- 1 apache apache 3895296 Oct 17 08:54 /u01/apache2/cache/aptmpboQGxr ########## Directory Listing ################## This is on a pretty busy web server but the number of files grows very quickly and so the htclean I have setup to run a limit cache size to 4GB is effectively useless as things reached 11GB until I started digging into where the "other 7GB" of storage went and discovered these "random" files. The only possibly interesting thing I'm doing is caching access to images and videos served from Tomcat servers using mod_jk (tomcat-connectors-1.2.37-src.tar.gz to be precise) and running mod_security but I don't see why those would cause the disk cache to leave stray files around. Again, sorry to disappoint the people who thought they'd fixed this.
Having just check I can confirm I do not find any of these temporary files on the non production servers which run exactly the same configuration and versions for Apache but experience only internal testing traffic not real users or real load. Load on the servers is at least 200 requests per second with about a quarter of those going through mod_disk_cache. This is significantly less that the servers can handle however so it's not a server overload causing this. Also I see not immediate indication that Apache threads or processes are crashing our causing these files to be left lying around (I'm assuming such crashes would appear in the main server error log where it's logs restart).
No indication anyone tried ruedigers patch
I have found one cause of this leaking. If a response ends abnornally, there is no original EOS bucket, and ap_die removes the cache_out filter, so the aptmp file is never cleaned up.
Undo spam change
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.