[SANTUARIO-459] Wrong usage of Integer.MAX_VALUE

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: Java 2.0.8, Java 2.1.0
• Environment: Java(TM) SE Runtime Environment (build 1.8.0_66-b17); IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64; Apache XML Security 2.0.7
• Flags: Patch
• Assignee: Colm O hEigeartaigh
• Reporter: Ronald Boettcher
• Created: 09/Nov/16 08:33
• Updated: 05/Dec/16 10:00
• Resolved: 09/Nov/16 11:16

Description

Most Java VMs are throwing a "Requested array size exceeds VM limit" OutOfMemoryException when using just Integer.MAX_VALUE or Integer.MAX_VALUE - 1

We are using Apache Santuario XML Encryption with large documents and encountered this problem.

I had to change it in org/apache/xml/security/utils/UnsyncByteArrayOutputStream.java.

The patch file content is as following (I'm missing an attachment button):

UnsyncByteArrayOutputStream.java.patch

*** src/main/java/org/apache/xml/security/utils/UnsyncByteArrayOutputStream.java.orig   2016-09-30 11:59:49.570187935 +0200
--- src/main/java/org/apache/xml/security/utils/UnsyncByteArrayOutputStream.java        2016-09-30 11:59:31.149998245 +0200
***************
*** 28,33 ****
--- 28,37 ----
  public class UnsyncByteArrayOutputStream extends OutputStream  {
  
      private static final int INITIAL_SIZE = 8192;
+     
+     // most Java VMs are throwing a "Requested array size exceeds VM limit" OutOfMemoryException
+     // when using just Integer.MAX_VALUE or Integer.MAX_VALUE - 1
+     private static final int VM_ARRAY_INDEX_MAX_VALUE = Integer.MAX_VALUE - 2;
  
      private byte[] buf;
      private int size = INITIAL_SIZE;
***************
*** 38,44 ****
      }
  
      public void write(byte[] arg0) {
!         if ((Integer.MAX_VALUE - pos) < arg0.length) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg0.length;
--- 42,48 ----
      }
  
      public void write(byte[] arg0) {
!         if ((VM_ARRAY_INDEX_MAX_VALUE - pos) < arg0.length) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg0.length;
***************
*** 50,56 ****
      }
  
      public void write(byte[] arg0, int arg1, int arg2) {
!         if ((Integer.MAX_VALUE - pos) < arg2) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg2;
--- 54,60 ----
      }
  
      public void write(byte[] arg0, int arg1, int arg2) {
!         if ((VM_ARRAY_INDEX_MAX_VALUE - pos) < arg2) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + arg2;
***************
*** 62,68 ****
      }
  
      public void write(int arg0) {
!         if (Integer.MAX_VALUE - pos == 0) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + 1;
--- 66,72 ----
      }
  
      public void write(int arg0) {
!         if (VM_ARRAY_INDEX_MAX_VALUE - pos == 0) {
              throw new OutOfMemoryError();
          }
          int newPos = pos + 1;
***************
*** 88,94 ****
              newSize = newSize << 1;
              // Deal with overflow
              if (newSize < 0) {
!                 newSize = Integer.MAX_VALUE;
              }
          }
          byte newBuf[] = new byte[newSize];
--- 92,98 ----
              newSize = newSize << 1;
              // Deal with overflow
              if (newSize < 0) {
!                 newSize = VM_ARRAY_INDEX_MAX_VALUE;
              }
          }
          byte newBuf[] = new byte[newSize];