Commons IO
  1. Commons IO
  2. IO-266

FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows 2003 Server 64-bit

      Description

      java.io.IOException: Insufficient system resources exist to complete the requested service
      at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
      at sun.nio.ch.FileDispatcher.pwrite(Unknown Source)
      at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
      at sun.nio.ch.IOUtil.write(Unknown Source)
      at sun.nio.ch.FileChannelImpl.write(Unknown Source)
      at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
      at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
      at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:813)
      at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:783)
      at org.test.igor.TestFileUtils.main(TestFileUtils.java:55)

      NOTE: the issue is cased by the function doCopyFile(File srcFile, File destFile, boolean preserveFileDate) using hardcoded data chunks of FIFTY_MB in the transferFrom() call.
      Reducing this chunk from 50M to 31M solves the issue for my situation (32M still fails).

      Here is a test program to reproduce the issue:

      package org.test.igor;

      import java.io.File;
      import java.io.FileInputStream;
      import java.io.FileOutputStream;
      import java.io.IOException;
      import java.nio.channels.FileChannel;

      import org.apache.commons.io.FileUtils;
      import org.apache.commons.io.IOUtils;

      public class TestFileUtils {

      public static void main(String[] args){

      File src = new File("D:\\2011.1-dev\\test
      test");
      File dest = new File("\\\\ismerek1
      Shared");

      String filename = "jdk-6u19-windows-x64.exe";

      File file = new File(src, filename);
      File toFile = new File(dest, filename);

      try

      { FileUtils.copyFile(file, toFile, true); System.out.println("Successful copy"); }

      catch (IOException e1)

      { e1.printStackTrace(); }

      }
      }

        Activity

        Hide
        Sebb added a comment -

        50MB does seem rather large as a default; I would have thought 16MB would be plenty?
        Perhaps we should make the default smaller.

        It would be possible to add new methods which allowed the buffer size to be provided; however this would mean adding about 7 new methods.

        Unfortunately changing the buffersize to a variable would be thread-hostile: if two threads set different values, there is a timing window between setting the value and starting the copy method. This cannot be avoided.

        It's a pity the class methods are all static ...

        One other possibility would be to allow the default to be set by using a System Property; this could be thread-safe, but the value could not be changed once set.

        Show
        Sebb added a comment - 50MB does seem rather large as a default; I would have thought 16MB would be plenty? Perhaps we should make the default smaller. It would be possible to add new methods which allowed the buffer size to be provided; however this would mean adding about 7 new methods. Unfortunately changing the buffersize to a variable would be thread-hostile: if two threads set different values, there is a timing window between setting the value and starting the copy method. This cannot be avoided. It's a pity the class methods are all static ... One other possibility would be to allow the default to be set by using a System Property; this could be thread-safe, but the value could not be changed once set.
        Hide
        Niall Pemberton added a comment -

        I think we should just reduce the size to fix the problem Igor found - so 30M is the nearest (rounded)

        Show
        Niall Pemberton added a comment - I think we should just reduce the size to fix the problem Igor found - so 30M is the nearest (rounded)
        Hide
        Arnaud BRAND added a comment -

        I attached a patch setting the chunk at 30MB since I'm also affected by this issue.

        Thanks.

        Show
        Arnaud BRAND added a comment - I attached a patch setting the chunk at 30MB since I'm also affected by this issue. Thanks.
        Hide
        Sebb added a comment -

        Changed to 30MB

        Show
        Sebb added a comment - Changed to 30MB
        Hide
        Gary Gregory added a comment -

        Closing, we released version 2.1.

        Show
        Gary Gregory added a comment - Closing, we released version 2.1.

          People

          • Assignee:
            Unassigned
            Reporter:
            Igor Smereka
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development