[DAFFODIL-1735] BLOB/CLOB - large object handles

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5.0
• Component/s: API, Back End, DFDL Language, Front End
• Assignee: Steve Lawrence
• Reporter: Mike Beckerle
• Created: 04/Apr/17 16:09
• Updated: 20/May/21 20:01
• Resolved: 05/Sep/19 14:28

Description

Binary large object, and character large object (aka BLOB/CLOB) support is needed.

Rather than transforming a large binary object into hexBinary or string, instead one annotates the DFDL schema to specify that a BLOB/CLOB handle is preferred. The Infoset then contains not the data, but a token/object that can be fed to an "open" API to access a stream of the data. The token/object is a small sized thing. This allows data formats like JPEG 2000 where payloads are often gigabytes.

Note: implementing this requires revising our ByteBufferDataInputStream layer into one that does not require the input data to fit in memory.

See article on Wiki about large objects:

https://opensource.ncsa.illinois.edu/confluence/pages/viewpage.action?pageId=54427836