Commons Compress
  1. Commons Compress
  2. COMPRESS-7

Creating TAR archive of an 8 GB file causes IOException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Win xp pro sp2; JDK 1.5.0_07

      Description

      Yesterday I reported a bug with extraction of TAR archives:
      https://issues.apache.org/jira/browse/SANDBOX-168

      Today I found that when I tried to creat a TAR archive of a file that is exactly 8 GB, that is,
      (8L * 1024L * 1024L * 1024L) - 1
      that it failed with the following strange error:

      Exception in thread "main" java.io.IOException: request to write '32768' bytes exceeds size in header of '-1' bytes
      at org.apache.commons.compress.archivers.tar.TarOutputStream.write(TarOutputStream.java:341)
      at bb.io.StreamUtil.transfer(StreamUtil.java:312)
      at bb.io.StreamUtil.transfer(StreamUtil.java:290)
      at bb.io.TarUtil.writeFileData(TarUtil.java:404)
      at bb.io.TarUtil.archive(TarUtil.java:382)
      at bb.io.TarUtil.archive(TarUtil.java:334)
      at bb.io.TarUtil$Test.test_archive_extract_fileLengthLimit(TarUtil.java:610)
      at bb.io.TarUtil$Test.main(TarUtil.java:542)

      The code that I used to try and create the TAR archive was the same described in that previous bug report, so I will not repeat it here.

      I was doing this in an attempt to discern if the org.apache.commons.compress TAR code has any file size limits, since classic TAR is limited to 8 BG files but org.apache.commons.compress supports GNU TAR long path names, so maybe it has no file size limits either. (There appears to be no documentation about this in the TAR code.)

      What is strange is that the code first DID apparently succeed in TARing a 10 GB file, albeit, when I tried to see if a third party piece of software (7-zip) would extract it, that program objected saying that it did not support the TAR format type encpuntered in the file.

      Since I need to use a file size limit that is consistent with other programs, I then tried to TAR a file of exactly 8 GB in size. This is when I encountered the above error.

        Activity

        Hide
        Jeff Crump added a comment -

        I was faced with a similar issue today. While the entry size is a long, the write() method takes an int – and if you get too big, you overflow. Here is the exception I got when trying to write 9GB entry

        java.io.IOException request to write '8192' bytes exceeds size in header of '1390354048' bytes
        at org.apache.commons.compress.tar.TarOutputStream.write(TarOutputStream.java:-3)
        at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:-3)
        at org.archivas.io.TransferSupport.writeBuffer(TransferSupport.java:-3)
        at org.archivas.io.TransferSupport.writeChunk(TransferSupport.java:-4)

        It looks like the Ant folks have taken care of this already?
        http://www.cenqua.com/fisheye/demo/browse/ant/src/main/org/apache/tools/tar/TarOutputStream.java

        Why is the m_size field a long, but the write() API take an int?

        Show
        Jeff Crump added a comment - I was faced with a similar issue today. While the entry size is a long, the write() method takes an int – and if you get too big, you overflow. Here is the exception I got when trying to write 9GB entry java.io.IOException request to write '8192' bytes exceeds size in header of '1390354048' bytes at org.apache.commons.compress.tar.TarOutputStream.write(TarOutputStream.java:-3) at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:-3) at org.archivas.io.TransferSupport.writeBuffer(TransferSupport.java:-3) at org.archivas.io.TransferSupport.writeChunk(TransferSupport.java:-4) It looks like the Ant folks have taken care of this already? http://www.cenqua.com/fisheye/demo/browse/ant/src/main/org/apache/tools/tar/TarOutputStream.java Why is the m_size field a long, but the write() API take an int?
        Hide
        Emmanuel Bourg added a comment -

        Fixed, TarOutputStream now uses long values internally.

        Show
        Emmanuel Bourg added a comment - Fixed, TarOutputStream now uses long values internally.

          People

          • Assignee:
            Unassigned
            Reporter:
            Sam Smith
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development