Dear Apache Developers, My name is Dan Kogai and I thank you for your great product. That definitely makes the world go round! I just found that Apache, both 1 and 2, does not support files that are larger than 2GB. Apache 2 does support it but only on (truly) 64-bit systems. On such systems (I would even say majority thereof) where large files are supported (off_t is 64-bit) but size_t is limited to 32-bit, it does not work. I browse the source and found that while apr_off_t is correctly used APR-wise, many calculations that contain apr_size_t variables led to incorrect values. The quick & dirty proof-of-concept patch below does fix the problem. Tested OK on FreeBSD 4, Mac OS X v10.3. Compiled on Linux kernel 2.4.9 but rerurns 400 with log entry "[client 127.0.0.1] (75)Value too large for defined data type: access to /dvd.dmg failed" As I said this is way too quick and dirty and I would appreciate if you guys fix that more properly and elegantly. It may sound ridiculous to handle such large files but I happen to be a member of Ring Server Project which hosts one of the most famous open source distribution and there appears DVD images that get influenced. Thank you in advance for your attention and support. Dan the Man with Too Many File Transfers --- httpd-2.0.49/srclib/apr/include/apr.h.in Sun Feb 29 03:41:32 2004 +++ httpd-2.0.49-x/srclib/apr/include/apr.h.in Tue May 11 21:27:31 2004 @@ -263,7 +263,7 @@ typedef @long_value@ apr_int64_t; typedef unsigned @long_value@ apr_uint64_t; -typedef @size_t_value@ apr_size_t; +typedef @off_t_value@ apr_size_t; typedef @ssize_t_value@ apr_ssize_t; typedef @off_t_value@ apr_off_t; typedef @socklen_t_value@ apr_socklen_t;
Can you precisely describe the issues you see on platforms with a 32-bit size_t and a 64-bit off_t?
Joe Orton, Thank you for your response. This is what happens; ls -l /ring/ftp/4gb-test.dmg -rw-r--r-- 1 root mirror 4699983872 May 11 22:45 /ring/ftp/4gb-test.dmg HEAD http://localhost/4gb-test.dmg 200 OK Connection: close Date: Fri, 14 May 2004 02:26:47 GMT Accept-Ranges: bytes Server: Apache Content-Length: 405016576 Content-Type: application/octet-stream ETag: "6-18241000-557d0100" Last-Modified: Tue, 11 May 2004 13:45:08 GMT Client-Date: Fri, 14 May 2004 02:26:46 GMT Client-Peer: 210.159.71.23:80 As you see, * Content-Length: shows only lower 32-bit thereof * File transfer fails accordingly (only 405,016,576 bytes of actual 4,699,983,872 bytes gets downloaded in the example above). Dan the Truncated Man
Interesting. That's a bug, but I can't reproduce it with HEAD on a 32-bit Linux using apr_off_t = off64_t, apr_size_t = size_t. $ HEAD http://localhost:8900/big/bigfile | grep Content-Length Content-Length: 3145728000 so it's something more subtle. Does the correct file size get logged to access_log if you GET the file? Does autoindex show the correct size for the file in a directory listing?
BTW, your HEAD test was with 2.0, I presume. 1.3 uses long rather than off_t for file sizes everywhere, so you're always limited by sizeof(long) in 1.3 even if sizeof(off_t) == 64.
> $ HEAD http://localhost:8900/big/bigfile | grep Content-Length > Content-Length: 3145728000 I don't think it's big enough. It's definitely larger than MAX_INT but well within UMAX_INT, which nicely fits in 32-bit. 3145728000 = 0xbb80_0000 < 0xffff_ffff FYI to make large files quickly (and sparsely), you can go like perl -e 'truncate shift, shift' file size or truncate -s size file if your platform has trucate(1). > BTW, your HEAD test was with 2.0, I presume. Correct. 1.3.x hardcodes them all "long" while 2.0.x uses apr_*. Dan the Truncated Man
> Does the correct file size get logged to access_log if you GET the file? With my (quick & dirty) patch, yes. w/o, no. > Does autoindex show the correct size for the file in a directory listing? Yes, with or without the patch. dvd.dmg 12-May-2004 00:07 4.4G Dan
Well, since off_t is signed any size >2Gb would fail, if at all... it works the same here for me using >4Gb sizes too (again, using HEAD with apr_off_t==off64_t, which should be equivalent to a FreeBSD system with apr_off_t==off_t where sizeof(off_t) == 8). So I don't know what bug you're seeing here, it would be great if you could debug it, i.e. work out if apr_stat() is determining the size correctly, and work on up...
> Well, since off_t is signed any size >2Gb would fail, if at all... > it works the same here for me using >4Gb sizes too The problem is sizeof(apr_off_t) > sizeof(apr_size_t) in those platforms while there are many places where apr_off_t objects are computed against apr_size_t objects. We already have learned that forcibily making apr_size_t 64-bit off-t fixes the problem (in some platforms). Oh, I have chenged the summery from "2GB" to "4GB" to be more precise. > it would be great if you could debug it Yikes. I know HOW it went wrong but WHERE to fix is another problem. Since it is size-related, that may even lead to changes in *.h, meaning API changes and that'way too much for me. > i.e. work out if apr_stat() is determining the size correctly, > and work on up... That's not the only problem. Anywhere apr_off_t is used in conjunction w/ apr_size_t are vulnerable. i.e apr_bucket_read(). Dan the Apache *User* (and love to stay that way)
In Mac OS X, I later found that while Content-Length: header was correct w/ the previous patch, the actual file transfer gets trucated. The following patch to emulate_sendfile() in server/core.c fixes that (YOU STILL NEED MY PREVIOUS PATCH). I wonder why FreeBSD did work. Maybe because it was tested w/ truncated (sparse) file. There on Mac OS X I have used the actual DVD image file for testing uI also applied byte-to-byte exhaustive file comparison as well as md5 sum to make sure the transferred file is identical to the original. That explains reason why my patch did not quite work on Linux. On Linux APR_HAS_SENDFILE is set and emulate_sendfile() is never used. Dan the Typedefed Man --- httpd-2.0.49/server/core.c Tue Mar 9 07:54:20 2004 +++ httpd-2.0.49-x/server/core.c Mon May 17 02:00:53 2004 @@ -2949,7 +2949,7 @@ apr_size_t length, apr_size_t *nbytes) { apr_status_t rv = APR_SUCCESS; - apr_int32_t togo; /* Remaining number of bytes in the file to send */ + apr_off_t togo; /* Remaining number of bytes in the file to send */ apr_size_t sendlen = 0; apr_size_t bytes_sent; apr_int32_t i;
Ah, nice work, yes, I see the issue here too with "EnableSendfile off". Can you try this patch - *without* your apr_off_t = size_t hack: http://www.apache.org/~jorton/ap_splitlfs.diff
> Can you try this patch - *without* your apr_off_t = size_t hack: Yay! Seems like it's working now. Both HEAD and GET works fine. Maybe we still need to check on (HTTP/1.1) partial transfers but this is a significant progress. Thank you so much! Dan the Untruncated Man
Joe, I am just curious if subbuckets also needs to turn off mmap (just to be sure). Here's the patch AGAINST YOURS that does that. Dan the Munmapped Man --- server/core.c Tue May 18 00:16:34 2004 +++ server/core.c.old Tue May 18 00:15:37 2004 @@ -3488,11 +3488,6 @@ while (fsize > AP_MAX_SENDFILE) { apr_bucket *ce; apr_bucket_copy(e, &ce); -#if APR_HAS_MMAP - if (d->enable_mmap == ENABLE_MMAP_OFF) { - (void)apr_bucket_file_enable_mmap(ce, 0); - } -#endif APR_BRIGADE_INSERT_TAIL(bb, ce); e->start += AP_MAX_SENDFILE; fsize -= AP_MAX_SENDFILE;
That patch is reversed... hmmm, probably. Actually it won't make any difference, since the file buckets are only mmap'ed if they are smaller than 4Mb, and here we're creating 16Mb buckets.
The patch is now committed for 2.1. Thanks for the report.