Bug 42696 - OutOfMemoryError using unzip Task
Summary: OutOfMemoryError using unzip Task
Status: RESOLVED FIXED
Alias: None
Product: Ant
Classification: Unclassified
Component: Core tasks (show other bugs)
Version: 1.7.0
Hardware: PC Windows XP
: P2 major with 1 vote (vote)
Target Milestone: 1.8.3
Assignee: Ant Notifications List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-06-19 06:33 UTC by Eickvonder
Modified: 2014-02-17 13:58 UTC (History)
2 users (show)



Attachments
Close zip entry input streams (1.44 KB, patch)
2009-09-15 10:45 UTC, David Schlosnagle
Details | Diff
a patch to rg/apache/tools/zip/ZipFile.java to end the inflater when the InflaterInputStream is closed (694 bytes, application/octet-stream)
2011-07-13 11:30 UTC, Mounir
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Eickvonder 2007-06-19 06:33:21 UTC
If I try to expand a rather huge zip file (in my case 42910 files, 783359kb 
uncompressed, 66906kb compressed) using the unzip task I get a 

java.lang.OutOfMemoryError: requested 37904 bytes for Chuck::new. Out of swap 
space?

My Java Options (Jdk 1.5.0_08 / Jdk 1.6_01) are

ANT_OPTS=-XX:NewRatio=2 -Xms512m -Xmx1024m -verbose:gc -XX:+PrintGCDetails

on a windows xp pro 32-bit system with a total of 2 GB memory available.
The problem now seems that this operation does not need much memory from the 
heap, so only one gc happens

[GC [DefNew: 139776K->17472K(157248K), 0.1300398 secs] 139776K->29555K(506816K),
 0.1301295 secs]

while the memory usage given by the windows task manager for the java process 
constantly increases towards 1GB before the process terminates.
If I reduce either Xms or increase NewRatio more gc happen and the memory usage 
as shown by the task manager never increases to more than 400 MB. At each gc it 
drops to 200 MB or less.
Decreasing Xms or increasing NewRatio is obviously a valid workaround for me, 
but nevertheless it is some kind of a memory leak here.

I even tried to find a cause for this (native) memory leak and it seems to be 
within org.apache.tools.zip.ZipFile. For me it seems that "new Inflater(true)"
causes this problem (though changing it simply to false will of course not 
work), because the problem still occurs if you comment out "extractFile" method 
in Expand.java such that nothing is really done except opening up input streams 
from the ZipFile for a number of ZipEntry objects.

So maybe this is in its core not a bug of ant, but a bug of the native java 
implementation for windows.
Comment 1 David Schlosnagle 2009-09-15 10:45:53 UTC
Created attachment 24269 [details]
Close zip entry input streams

I am encountering a similar issue using the unzip task on IBM AIX using Ant 1.6.5 and Java 1.5 and 1.6. The error I see is: "java.lang.OutOfMemoryError: ZIP004:OutOfMemoryError, MEM_ERROR in inflateInit2". We are performing several unzip tasks, including extracting a large WAR (~200MB) from a large EAR (~500MB). We are using ANT_OPTS="-Xms384M -Xmx1024M"

I looked into the org.apache.tools.ant.taskdefs.Expand.expandFile(FileUtils, File, File) code and it does not appear to be closing the input stream (and in turn the java.util.zip.Inflater instance for the returned java.util.zip.InflaterInputStream) for the zip entry, so I'm wondering if this is part of the problem. I've attached a patch to close each zip input stream.
Comment 2 Stefan Bodewig 2009-09-15 21:13:52 UTC
a modified version of the patch went into svn revision 815599 - it would be great if you could verify this fixed the original problem as well.

So far I expected the ZipFile code to be the cultprit, in particular I assumed the internal bookkeeping was too much for an archive that big.
Comment 3 Michael Ebi 2009-12-08 01:52:18 UTC
Stefan your Bugfix doesnt work for me. I get still an java.lang.OutOfMemoryError on a jar file with 27MB and 15.000 entries.
It disappears when inserting an trace.
log("expand "+ze.getName(), Project.MSG_INFO);
before the try block.
Comment 4 Stefan Bodewig 2009-12-09 09:40:44 UTC
(In reply to comment #3)
> Stefan your Bugfix doesnt work for me. I get still an
> java.lang.OutOfMemoryError on a jar file with 27MB and 15.000 entries.
> It disappears when inserting an trace.
> log("expand "+ze.getName(), Project.MSG_INFO);
> before the try block.

The log statement probably creates more temporary objects and adds pressure to the garbage collector, thus making the problem go away.

From the original description and David's analysis it seems as if java.util.zip.InflaterInputStream wasnt releasing all resources, even once closed.

Of course we can always add a debug statement if it helps avoiding the problem.  svn revision 888888 (cool revision number, isn't it?)
Comment 5 Mounir 2011-07-13 11:30:47 UTC
Created attachment 27282 [details]
a patch to rg/apache/tools/zip/ZipFile.java to end the inflater when the InflaterInputStream is closed

The OutOfMemoryError still happens with the head version. Although the fix to this bug was a call to the close() of the zip file input stream after each expand, the Inflater's native end() is never called, and thus memory is never released. This is because we are not using the InflaterInputStream's default Inflater, we are passing a new instance (you can check that InflaterInputStream at close ends the inflater only if it is the default inflater, and not passed from outside). I have patched ZipFile (check attached) to explicitly end the inflater when the input stream is closed, and the problem is resolved
Comment 6 Stefan Bodewig 2011-07-13 13:46:26 UTC
Thanks, patch applied as svn revision 1146020 

It turns out that Harmony's code will invoke end(), but OpenJDK's does not.

I've also opened https://issues.apache.org/jira/browse/COMPRESS-139 and will port your patch to commons-compress as well.  Thanks!

BTW, if you want your full name to appear in the contributors list, jsut drop my a note.