I have identified the root cause of the issue and I have a solution.
JVM experiences java.lang.OutOfMemoryError during ivy:publish of a file greater than a few hundred MB.
HttpURLConnection object as configured in org.apache.ivy.util.url.BasicURLHandler buffers the data before sending so as to calculate the content-length header value. This alone is bad however is likely aggravated by the default array resizing algorythm of the ByteArrayOutputStream used to buffer the data.
The buffer starts small having a 32 byte capacity. When written to, it fills up and eventually it reaches maximum capacity. At this time a new buffer of twice the size is allocated. The content of the original buffer is copied to the new buffer and the old buffer is garbage collected. Lather, rinse, repeat.
This works fine for small amounts of data however this is a major problem when talking about medium or large amounts of data. As the buffer size approaches 50% of the amount of free heap space, there will no longer be enough free RAM to allocate the new buffer. For example, if a 512MB buffer requires even one more byte of capacity, and there is one byte less than 1024MB free, then an attempt by ByteArrayOutputStream to allocate a new 1024MB buffer will fail with an OOME. This is a tragic waste of memory, especially if the content-length is already known and buffering is not even required (which is always the case for Ivy).
Partial/Failed Solution Using HttpClient:
For one reason or another, possibly to fix this issue, a reflection invocation is made by org.apache.ivy.util.url.URLHandlerRegistry for org.apache.commons.httpclient.HttpClient. Comments about Ivy-1197 instruct users to place HttpClient jar on the classpath (lib dir of Ant) to solve the OOME issue. Indeed, HttpClient (even the ancient version from 2005 used by Ivy) does not buffer the content and thus could avoid the OOME entirely, however there are three major problems with this solution.
The first problem is that the trial invocation of HttpClient issued to detect availablility on the classpath fails rather silently by logging a vague message, and only if Ant is invoked in verbose mode. So somebody might drop the jar into the lib, see nothing happen, and wonder why.
The second problem is that the trial invocation of HttpClient does indeed fail if all one does is drop the httpclient jar into the lib dir. This is because HttpClient requires two additional jars in order to instantiate: commons-logging and commons-codec.
Adding these jars successfully triggers the Ivy code which substitutes the Apache HttpClient based org.apache.ivy.util.url.HttpClientHandler for the problematic URLConnectionHandler based org.apache.ivy.util.url.BasicURLConnectionHandler. But there is one more problem.
The third problem is as follows: Apache docs in the HttpClient performance guide describe how to stream a request using a custom RequestEntity object. This object is capable of restarting the stream in the event of an interruption or an authentication request. They provide sample code. This code was copied into the Ivy HttpClientHandler however the block that writes directly to the OutputStream (without buffering) has been replaced by a call to the same method that URLConnectionHandler calls which buffers all of the data. The net effect is that the OOME persists because nothing about the upload has changed - the data is still buffered in a ByteArrayOutputStream.
In HttpClientHandler.FileRequestEntity.writeRequest(OutputStream) replace this line:
FileUtil.copy(instream, out, null, false);
with the original Apache sample code (slightly refactored to match the class):
byte buffer = new byte[64*1024];
while ((length = instream.read(buffer)) != -1)
out.write(buffer, 0, length);
Then recompile Ivy. I recompiled using JDK 1.6 because that is the oldest VM around here.
This fix only works when all of the following libraries are located in $ANT_HOME/lib directory:
Those libraries are available in the apache-ivy-2.3.0-bin-with-deps distribution.