Uploaded image for project: 'Daffodil'
  1. Daffodil
  2. DAFFODIL-2194

buffered data output stream has a chunk limit of 2GB

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.5.0
    • Back End
    • None

    Description

      A buffered data outupt stream is backed by a growable ByteArrayOutputStream, which can only grow to 2GB in size. So if we ever try to write more than 2GB to a buffered output stream during unparse (very possible with large blobs), we'll get an OutOfMemoryError.

      One potential solution is to be aware of the size of a ByteArrayOutputStream when buffering output and automatically create a split when it gets to 2GB in sizes. This will still require a ton of memory since we're buffering these in memoary, but we'll at least be able to unparse more than 2GB of continuous data.

      Note that we should still be able to unparse more than 2GB of data total, as long as there so single buffer that's more than 2GB.

      Attachments

        Activity

          People

            dfthompson Dave Thompson
            slawrence Steve Lawrence
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: