Uploaded image for project: 'Daffodil'
  1. Daffodil
  2. DAFFODIL-2425

unparse fail with large blob

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Back End, CLI
    • None

    Description

      I am running large NITF files against Daffodil. I can parse the large file, but Daffodil crashes on the unparse. My casual inspection suggests that a putFile function needs to accept a long parameter instead of an int.

      I am running Apache Daffodil (incubating) 3.0.0-SNAPSHOT 

      As per https://issues.apache.org/jira/browse/DAFFODIL-2401 I modified the NITF DFDL Schema to use blobs (instead of HexBinary). The modified Schema (and a supporting jpeg Schema) is attached here as I have not yet made a pull request to the NITF DFDL project to have the modifications placed into the main repo.

      Be aware that the zipped, compressed test file 'test4G.ntf.zip.Z' will expand to a 4GB file.

      To parse the NITF file:

      ~/Daffodil/daffodil-cli/target/universal/stage/bin/daffodil parse -I xml -o xxx -s nitfx.dfdl.xsd test4G.ntf

      and then to unparse

      ~/Daffodil/daffodil-cli/target/universal/stage/bin/daffodil unparse -I xml -o xxx.nitf -s nitfx.dfdl.xsd xxx

      and after some warnings, it's off into the weeds:

      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
      !! An unexpected exception occurred. This is a bug! !!
      !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

      Please report this bug and help us fix it:

      https://daffodil.apache.org/community/#issue-tracker

      Please include the following exception, the command you
      ran, and any input, schema, or tdml files used that led
      to this bug.

      org.apache.daffodil.exceptions.Abort: Invariant broken: nBytesPut.==(nBytes)
      org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
      org.apache.daffodil.io.DirectOrBufferedDataOutputStream.deliverContent(DirectOrBufferedDataOutputStream.scala:891)
      org.apache.daffodil.io.DirectOrBufferedDataOutputStream$.org$apache$daffodil$io$DirectOrBufferedDataOutputStream$$deliverBufferContent(DirectOrBufferedDataOutputStream.scala:1022)
      org.apache.daffodil.io.DirectOrBufferedDataOutputStream.setFinished(DirectOrBufferedDataOutputStream.scala:457)
      at org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
      at org.apache.daffodil.io.DirectOrBufferedDataOutputStream.deliverContent(DirectOrBufferedDataOutputStream.scala:891)
      at org.apache.daffodil.io.DirectOrBufferedDataOutputStream$.org$apache$daffodil$io$DirectOrBufferedDataOutputStream$$deliverBufferContent(DirectOrBufferedDataOutputStream.scala:1022)
      at org.apache.daffodil.io.DirectOrBufferedDataOutputStream.setFinished(DirectOrBufferedDataOutputStream.scala:457)
      at org.apache.daffodil.processors.Suspension.runSuspension(Suspension.scala:89)
      at org.apache.daffodil.processors.Suspension.runSuspension$(Suspension.scala:81)
      at org.apache.daffodil.processors.unparsers.OnlyPaddingUnparserSuspendableOperation.runSuspension(SpecifiedLength2.scala:616)
      at org.apache.daffodil.processors.SuspensionTracker.evalSuspensionQueue(SuspensionTracker.scala:110)
      at org.apache.daffodil.processors.SuspensionTracker.requireFinal(SuspensionTracker.scala:80)
      at org.apache.daffodil.processors.unparsers.UStateMain.evalSuspensions(UState.scala:620)
      at org.apache.daffodil.processors.DataProcessor.unparse(DataProcessor.scala:572)
      at org.apache.daffodil.processors.DataProcessor.unparse(DataProcessor.scala:545)
      at org.apache.daffodil.Main$.$anonfun$run$20(Main.scala:1222)
      at org.apache.daffodil.util.Timer$.getTimeResult(Timer.scala:78)
      at org.apache.daffodil.util.Timer$.getResult(Timer.scala:37)
      at org.apache.daffodil.Main$.run(Main.scala:1222)
      at org.apache.daffodil.Main$.main(Main.scala:1464)
      at org.apache.daffodil.Main.main(Main.scala)

      Attachments

        1. test4G.ntf.zip.Z
          6 kB
          Brent Nordin
        2. nitfx.dfdl.xsd
          71 kB
          Brent Nordin
        3. jpeg.dfdl.xsd
          43 kB
          Brent Nordin

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            bnordin Brent Nordin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment