[DAFFODIL-934] Streaming parser: Need to stream input data in, and infoset out to handle arbitrarily large data.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: s13
• Fix Version/s: 3.0.0
• Component/s: Performance
• Assignee: Steve Lawrence
• Reporter: Mike Beckerle
• Created: 08/Apr/14 22:20
• Updated: 24/Sep/20 20:15
• Resolved: 04/Sep/20 17:28

Description

Currently Daffodil requires that all incoming data fit in one java.nio.ByteBuffer. A separate issue (DFDL-881) is about allowing > 4GB files, but data sizes would still be limited by available address space.

A streaming approach has great advantages. It requires that the input can be streamed in (e.g., from a java.io.InputStream), but also requires that the DFDL Infoset can be streamed out. (Think SAX parser 'events' coming out). This is complicated by the DFDL notion of points of uncertainty. E.g., until a choice branch has been resolved none of the elements on any branch can be emitted since "backtracking" may invalidate them.

Issue Links

• is duplicated by: DAFFODIL-1799 Enable data streaming in the CLI