[IO-400] IOUtils: add support for copying from large byte buffers

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 2.5
• Assignee: Unassigned
• Reporter: Sebb
• Created: 29/May/13 23:55
• Updated: 17/Oct/13 00:25
• Resolved: 17/Oct/13 00:25

Description

Trying to write a large byte array to a FileOutputStream may cause OOME.

This is because such output requires the use of native code, and native code may need to copy the array in order to access it safely, see:

http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp1265

It might therefore be useful to have a method which writes from the byte array in chunks. One can create a ByteArrayInputStream from the input, and then use one of the copy() methods, but that creates an unnecessary array buffer (albeit only 4k).

There are already write methods which copy byte[] to OutputStream and char[] to Writer.

Some or all of these could be converted to use chunked output, or there could be new methods to implement the chunking.

Here is a sample implementation of a stand-alone method:

public static void writeChunked(byte[] data, OutputStream output) throws IOException {
    int bytes = data.length;
    int offset = 0;
    while(bytes > 0) {
        int chunk = Math.min(bytes, DEFAULT_BUFFER_SIZE);
        output.write(data, offset, chunk);
        bytes -= chunk;
        offset += chunk;
    }
}

Issue Links

• duplicates: IO-382 Chunked IO for large arrays