[DAFFODIL-2425] unparse fail with large blob

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Back End, CLI
• Assignee: Unassigned
• Reporter: Brent Nordin
• Created: 27/Oct/20 00:20
• Updated: 17/Nov/20 13:41
• Resolved: 27/Oct/20 17:34

Description

I am running large NITF files against Daffodil. I can parse the large file, but Daffodil crashes on the unparse. My casual inspection suggests that a putFile function needs to accept a long parameter instead of an int.

I am running Apache Daffodil (incubating) 3.0.0-SNAPSHOT 

As per https://issues.apache.org/jira/browse/DAFFODIL-2401 I modified the NITF DFDL Schema to use blobs (instead of HexBinary). The modified Schema (and a supporting jpeg Schema) is attached here as I have not yet made a pull request to the NITF DFDL project to have the modifications placed into the main repo.

Be aware that the zipped, compressed test file 'test4G.ntf.zip.Z' will expand to a 4GB file.

To parse the NITF file:

~/Daffodil/daffodil-cli/target/universal/stage/bin/daffodil parse -I xml -o xxx -s nitfx.dfdl.xsd test4G.ntf

and then to unparse

~/Daffodil/daffodil-cli/target/universal/stage/bin/daffodil unparse -I xml -o xxx.nitf -s nitfx.dfdl.xsd xxx

and after some warnings, it's off into the weeds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! An unexpected exception occurred. This is a bug! !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Please report this bug and help us fix it:

https://daffodil.apache.org/community/#issue-tracker

Please include the following exception, the command you
ran, and any input, schema, or tdml files used that led
to this bug.

org.apache.daffodil.exceptions.Abort: Invariant broken: nBytesPut.==(nBytes)
org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
org.apache.daffodil.io.DirectOrBufferedDataOutputStream.deliverContent(DirectOrBufferedDataOutputStream.scala:891)
org.apache.daffodil.io.DirectOrBufferedDataOutputStream$.org$apache$daffodil$io$DirectOrBufferedDataOutputStream$$deliverBufferContent(DirectOrBufferedDataOutputStream.scala:1022)
org.apache.daffodil.io.DirectOrBufferedDataOutputStream.setFinished(DirectOrBufferedDataOutputStream.scala:457)
at org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
at org.apache.daffodil.io.DirectOrBufferedDataOutputStream.deliverContent(DirectOrBufferedDataOutputStream.scala:891)
at org.apache.daffodil.io.DirectOrBufferedDataOutputStream$.org$apache$daffodil$io$DirectOrBufferedDataOutputStream$$deliverBufferContent(DirectOrBufferedDataOutputStream.scala:1022)
at org.apache.daffodil.io.DirectOrBufferedDataOutputStream.setFinished(DirectOrBufferedDataOutputStream.scala:457)
at org.apache.daffodil.processors.Suspension.runSuspension(Suspension.scala:89)
at org.apache.daffodil.processors.Suspension.runSuspension$(Suspension.scala:81)
at org.apache.daffodil.processors.unparsers.OnlyPaddingUnparserSuspendableOperation.runSuspension(SpecifiedLength2.scala:616)
at org.apache.daffodil.processors.SuspensionTracker.evalSuspensionQueue(SuspensionTracker.scala:110)
at org.apache.daffodil.processors.SuspensionTracker.requireFinal(SuspensionTracker.scala:80)
at org.apache.daffodil.processors.unparsers.UStateMain.evalSuspensions(UState.scala:620)
at org.apache.daffodil.processors.DataProcessor.unparse(DataProcessor.scala:572)
at org.apache.daffodil.processors.DataProcessor.unparse(DataProcessor.scala:545)
at org.apache.daffodil.Main$.$anonfun$run$20(Main.scala:1222)
at org.apache.daffodil.util.Timer$.getTimeResult(Timer.scala:78)
at org.apache.daffodil.util.Timer$.getResult(Timer.scala:37)
at org.apache.daffodil.Main$.run(Main.scala:1222)
at org.apache.daffodil.Main$.main(Main.scala:1464)
at org.apache.daffodil.Main.main(Main.scala)