If I try to expand a rather huge zip file (in my case 42910 files, 783359kb uncompressed, 66906kb compressed) using the unzip task I get a java.lang.OutOfMemoryError: requested 37904 bytes for Chuck::new. Out of swap space? My Java Options (Jdk 1.5.0_08 / Jdk 1.6_01) are ANT_OPTS=-XX:NewRatio=2 -Xms512m -Xmx1024m -verbose:gc -XX:+PrintGCDetails on a windows xp pro 32-bit system with a total of 2 GB memory available. The problem now seems that this operation does not need much memory from the heap, so only one gc happens [GC [DefNew: 139776K->17472K(157248K), 0.1300398 secs] 139776K->29555K(506816K), 0.1301295 secs] while the memory usage given by the windows task manager for the java process constantly increases towards 1GB before the process terminates. If I reduce either Xms or increase NewRatio more gc happen and the memory usage as shown by the task manager never increases to more than 400 MB. At each gc it drops to 200 MB or less. Decreasing Xms or increasing NewRatio is obviously a valid workaround for me, but nevertheless it is some kind of a memory leak here. I even tried to find a cause for this (native) memory leak and it seems to be within org.apache.tools.zip.ZipFile. For me it seems that "new Inflater(true)" causes this problem (though changing it simply to false will of course not work), because the problem still occurs if you comment out "extractFile" method in Expand.java such that nothing is really done except opening up input streams from the ZipFile for a number of ZipEntry objects. So maybe this is in its core not a bug of ant, but a bug of the native java implementation for windows.
Created attachment 24269 [details] Close zip entry input streams I am encountering a similar issue using the unzip task on IBM AIX using Ant 1.6.5 and Java 1.5 and 1.6. The error I see is: "java.lang.OutOfMemoryError: ZIP004:OutOfMemoryError, MEM_ERROR in inflateInit2". We are performing several unzip tasks, including extracting a large WAR (~200MB) from a large EAR (~500MB). We are using ANT_OPTS="-Xms384M -Xmx1024M" I looked into the org.apache.tools.ant.taskdefs.Expand.expandFile(FileUtils, File, File) code and it does not appear to be closing the input stream (and in turn the java.util.zip.Inflater instance for the returned java.util.zip.InflaterInputStream) for the zip entry, so I'm wondering if this is part of the problem. I've attached a patch to close each zip input stream.
a modified version of the patch went into svn revision 815599 - it would be great if you could verify this fixed the original problem as well. So far I expected the ZipFile code to be the cultprit, in particular I assumed the internal bookkeeping was too much for an archive that big.
Stefan your Bugfix doesnt work for me. I get still an java.lang.OutOfMemoryError on a jar file with 27MB and 15.000 entries. It disappears when inserting an trace. log("expand "+ze.getName(), Project.MSG_INFO); before the try block.
(In reply to comment #3) > Stefan your Bugfix doesnt work for me. I get still an > java.lang.OutOfMemoryError on a jar file with 27MB and 15.000 entries. > It disappears when inserting an trace. > log("expand "+ze.getName(), Project.MSG_INFO); > before the try block. The log statement probably creates more temporary objects and adds pressure to the garbage collector, thus making the problem go away. From the original description and David's analysis it seems as if java.util.zip.InflaterInputStream wasnt releasing all resources, even once closed. Of course we can always add a debug statement if it helps avoiding the problem. svn revision 888888 (cool revision number, isn't it?)
Created attachment 27282 [details] a patch to rg/apache/tools/zip/ZipFile.java to end the inflater when the InflaterInputStream is closed The OutOfMemoryError still happens with the head version. Although the fix to this bug was a call to the close() of the zip file input stream after each expand, the Inflater's native end() is never called, and thus memory is never released. This is because we are not using the InflaterInputStream's default Inflater, we are passing a new instance (you can check that InflaterInputStream at close ends the inflater only if it is the default inflater, and not passed from outside). I have patched ZipFile (check attached) to explicitly end the inflater when the input stream is closed, and the problem is resolved
Thanks, patch applied as svn revision 1146020 It turns out that Harmony's code will invoke end(), but OpenJDK's does not. I've also opened https://issues.apache.org/jira/browse/COMPRESS-139 and will port your patch to commons-compress as well. Thanks! BTW, if you want your full name to appear in the contributors list, jsut drop my a note.