Uploaded image for project: 'Commons IO'
  1. Commons IO
  2. IO-650

Improve IOUtils performance by increasing DEFAULT_BUFFER_SIZE

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0
    • Fix Version/s: 2.6
    • Component/s: None
    • Labels:
      None

      Description

      IOUtils has a 4096B default buffer size that is used by copy() methods (and will be used for contentEquals methods when IO-649 is pulled).  This number should be updated to 8192 for a few reasons:

      1. It has a big improvement in performance in my micro-benchmark.  I tested both copy() and contentEquals() with 4K and 8K buffers.  This was done on a Late 2019 Macbook Pro (2.8GHz i7) with a 128MB file loaded into the OS buffer cache.  See below for the test harness used.  Past 8K performance does improve but it begins to experience some diminishing returns and could lead to excessive memory allocation.
      2. It mirrors the default buffer size of java.io.Buffered* classes.  This makes buffer sizing consistent regardless of if it is being done internally in the method or externally via a Buffered*.  These classes are used internally in IOUtils as well so the buffer size is not unreasonable.

       

      For copy():

      Metric 8K Buffer Millisecond 4K Buffer Millisecond Performance Improvement
      AVG 44.2853417 64.2600679 0.31084197
      P50 42.692406 62.371984 0.31551951
      P90 49.5538826 68.4303876 0.27584975
      P99 62.8831473 89.759114 0.29942326
      P100 102.563615 177.143364 0.42101351

       

      For contentEquals() with IO-649:

      Metric 8K Buffer Millisecond 4K Buffer Millisecond Performance Improvement
      AVG 81.5009567 128.497828 0.36574059
      P50 78.517749 124.191476 0.36776861
      P90 89.9172708 136.779763 0.34261276
      P99 125.814333 183.881989 0.31578762
      P100 308.936585 559.611217 0.44794426

      public static void main(String[] args) throws Exception {
           NullOutputStream nos = NullOutputStream.NULL_OUTPUT_STREAM;
           for (int i = 0; i < 1000; i++) {
               InputStream fis = new FileInputStream(new File("/tmp/random_data"));
               long start = System.nanoTime();
               IOUtils.copy(fis, nos);
               long defaultCopyTime = System.nanoTime() - start;

               fis = new FileInputStream(new File("/tmp/random_data"));

               start = System.nanoTime();
               IOUtils.copy(fis, nos, 8192);
               long bufferSizeSpecifiedCopyTime = System.nanoTime() - start;

               System.out.println(bufferSizeSpecifiedCopyTime + "\t" + defaultCopyTime);
           }
      }

      public static void main(String[] args) throws Exception {
          for (int i = 0; i < 1000; i++) {
              InputStream fis = new FileInputStream(new File("/tmp/random_data"));
              InputStream fis2 = new FileInputStream(new File("/tmp/random_data"));

              long start = System.nanoTime();
              IOUtils.contentEquals(fis, fis2);
              long defaultContentEqualsTime = System.nanoTime() - start;

              fis = new FileInputStream(new File("/tmp/random_data"));
              fis2 = new FileInputStream(new File("/tmp/random_data"));

               start = System.nanoTime();
               IOUtils.contentEquals(fis, fis2, 8192);
               long bufferSizeSpecifiedContentEqualsTime = System.nanoTime() - start;

               System.out.println(bufferSizeSpecifiedContentEqualsTime + "\t" + defaultContentEqualsTime);

          }

      }

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              blounsbury Brett Lounsbury
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: