[SANTUARIO-334] UnsyncByteArrayOutputStream hangs on messages larger 512 MB

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Java 1.5.2
• Fix Version/s: Java 1.4.8, Java 1.5.3
• Component/s: Java
• Security Level: Public (Public issues, viewable by everyone)
• Environment: Windows 7
• Assignee: Colm O hEigeartaigh
• Reporter: Torsten Keim
• Created: 31/Jul/12 07:05
• Updated: 06/Nov/12 13:30
• Resolved: 27/Oct/12 10:53

Description

If we check the signature of a message larger than 512 MB, the method expandSize(int newPos) of class org.apache.xml.security.utils.UnsyncByteArrayOutputStream goes in an endless loop, i.e. it hangs.
The following lines show the endless loop:
while (newPos > newSize) {
newSize = newSize << 2;
}
Initially the size is 8 KB and after several bit shifts by 2 (equal to multiplication with 4) the size is 512 MB. If 512 MB is multiplied with 4 it comes to an overflow: 512*1024*1024 << 2 = -2147483648, which is Integer.MIN_VALUE.
One solution would be to change the used types from int to long.

If we stay with datatype int, we could improve the implementation:
while (newPos > newSize) {
newSize = (newSize < (512*1024*1024)) ? (newSize << 2) :Integer.MAX_VALUE ;
}
This would work with message up to 2 GB. Messages greater than 2 GB would still fail.