[DAFFODIL-1933] Performance regression with new Base64/Layering changes

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Back End
• Assignee: Dave Thompson
• Reporter: Steve Lawrence
• Created: 04/May/18 12:24
• Updated: 25/May/18 11:51
• Resolved: 23/May/18 11:45

Description

Nightly tests showed a fairly drastic performance regression (50% reduction in speed) in commit 1ea2290f28: All properties for Base64/layering implemented.

Skimming the code, one change jumped out at me as a likely cause in ByteBufferDataInputStream.scala:

@@ -91,7 +90,13 @@ object ByteBufferDataInputStream {
       case _ => {
         // copy the contents of the stream into an array of bytes
         val bos = new ByteArrayOutputStream
-        IOUtils.copy(in, bos)
+        var b: Int = 0
+        while ({
+          b = in.read()
+          b != -1
+        }) {
+          bos.write(b)
+        }
         bos.flush()
         bos.close()
         in.close()

This changes the copy from a bulk copy to a byte-by-byte copy, which appears to have pretty drastic performance. A few simple quick tests shows that reverting this change recovers all performance losses.

Mike, can you confirm if this change can be reverted, or if there was some other reason related to layering to switch to a byte-by-byte copy?