Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 1.13
    • Archivers
    • None

    Description

      ZipFile (and SevenZFile) currently require a File argument, but it would be nice to support in-memory byte buffers rather than requiring temp files. Perhaps create a new SeekableInputStream class (or SeekableDataInput interface) and add corresponding constructors.

      For convenience, perhaps also add a utility class that wraps a ByteBuffer and/or byte[] and implements the new interface.

      (The sevenz package appears to have a similar limitation, so it might make sense to add the support there at the same time, but I personally don't have a need for that.)

      Attachments

        Activity

          I wrote a patch precisely for this a few weeks ago (attached), but got no feedback when I emailed it to the dev mailing list, so have been reluctant to commit it.

          Also the next major version of compress is supposed to support seekable streams, but development on it seems to have been abandoned?

          damjan Damjan Jovanovic added a comment - I wrote a patch precisely for this a few weeks ago (attached), but got no feedback when I emailed it to the dev mailing list, so have been reluctant to commit it. Also the next major version of compress is supposed to support seekable streams, but development on it seems to have been abandoned?
          bjkail Brett Kail added a comment -

          The patch seems to be missing the new SeekableInputStream and SeekableFileInputStream classes, but the changes seem to satisfy the request, so they seem reasonable to me .

          bjkail Brett Kail added a comment - The patch seems to be missing the new SeekableInputStream and SeekableFileInputStream classes, but the changes seem to satisfy the request, so they seem reasonable to me .
          bodewig Stefan Bodewig added a comment -

          The comment is going to be almost completely off-topic for the issue at hand. The only relevant bit: The main problem is that SeekableInputStream requires Java7 which might make the next release incompatible enough with 1.10 to warrant a new major release (officially Compress 1.x supports Java5).

          damjan I don't remember seeing you post to the dev list, must have missed it (or it never reached the list?).

          The 2.x branch https://git-wip-us.apache.org/repos/asf?p=commons-compress.git;a=shortlog;h=refs/heads/compress-2.0 is stalled more than abandoned, I'm not sure this makes a difference, though. IMHO the ideas are viable but I've run out of time (or rather spent my OSS time slices elsewhere). It would be a lot easier to find motivation picking it up again if other people would join.

          bodewig Stefan Bodewig added a comment - The comment is going to be almost completely off-topic for the issue at hand. The only relevant bit: The main problem is that SeekableInputStream requires Java7 which might make the next release incompatible enough with 1.10 to warrant a new major release (officially Compress 1.x supports Java5). damjan I don't remember seeing you post to the dev list, must have missed it (or it never reached the list?). The 2.x branch https://git-wip-us.apache.org/repos/asf?p=commons-compress.git;a=shortlog;h=refs/heads/compress-2.0 is stalled more than abandoned, I'm not sure this makes a difference, though. IMHO the ideas are viable but I've run out of time (or rather spent my OSS time slices elsewhere). It would be a lot easier to find motivation picking it up again if other people would join.

          It was a 30 September 2015 email entitled "[COMPRESS] generalize RandomAccessFile in ZipFile and SevenZFile", link here:
          http://www.mail-archive.com/dev@commons.apache.org/msg49795.html

          I found the names of methods in 2.x were unintuitive, and how it's supposed to be used is confusing. Is there any design notes or documentation?

          damjan Damjan Jovanovic added a comment - It was a 30 September 2015 email entitled " [COMPRESS] generalize RandomAccessFile in ZipFile and SevenZFile", link here: http://www.mail-archive.com/dev@commons.apache.org/msg49795.html I found the names of methods in 2.x were unintuitive, and how it's supposed to be used is confusing. Is there any design notes or documentation?

          I think it would be fine to bump the Java req to Java 7 at this point.

          ggregory Gary D. Gregory added a comment - I think it would be fine to bump the Java req to Java 7 at this point.
          bodewig Stefan Bodewig added a comment -

          We might be better off with discussing this on the dev list

          damjan I'll look into the patch soonish, but likely not before Sunday. There is no design doc for the 2.x branch, just a couple of mail threads to the dev list from late 2013, early 2014 (I think the first message I sent was http://mail-archives.apache.org/mod_mbox/commons-dev/201312.mbox/%3C877gbf4yyg.fsf%40v35516.1blu.de%3E

          bodewig Stefan Bodewig added a comment - We might be better off with discussing this on the dev list damjan I'll look into the patch soonish, but likely not before Sunday. There is no design doc for the 2.x branch, just a couple of mail threads to the dev list from late 2013, early 2014 (I think the first message I sent was http://mail-archives.apache.org/mod_mbox/commons-dev/201312.mbox/%3C877gbf4yyg.fsf%40v35516.1blu.de%3E
          bodewig Stefan Bodewig added a comment -

          It took me way longer than I expected, sorry about that.

          Before I start making noise on the dev list, damjan does your code actually require Java7? Neither your patch here, nor the one sent to the dev list contain the Seekable*Stream classes themselves.

          bodewig Stefan Bodewig added a comment - It took me way longer than I expected, sorry about that. Before I start making noise on the dev list, damjan does your code actually require Java7? Neither your patch here, nor the one sent to the dev list contain the Seekable*Stream classes themselves.

          Sorry. Here is the patch with all the files.

          damjan Damjan Jovanovic added a comment - Sorry. Here is the patch with all the files.
          bodewig Stefan Bodewig added a comment -

          Thanks

          AFAICT there isn't anything requiring Java7 in your patch - I had assumed you'd be using SeekableByteChannel. Sorry for the noise.

          As usual we'll likely quibble about a few names (getFilePosition in SeekableInputStream) and I'm not sure whether it wouldn't be better to turn SeekableInputStream into an interface - haven't checked whether ZipFile/SevenZFile need any methods of InputStream that are not part of SeekableInputStream. But this can be easily resolved on the list.

          I'd be happy if you applied the patch.

          bodewig Stefan Bodewig added a comment - Thanks AFAICT there isn't anything requiring Java7 in your patch - I had assumed you'd be using SeekableByteChannel. Sorry for the noise. As usual we'll likely quibble about a few names (getFilePosition in SeekableInputStream) and I'm not sure whether it wouldn't be better to turn SeekableInputStream into an interface - haven't checked whether ZipFile/SevenZFile need any methods of InputStream that are not part of SeekableInputStream. But this can be easily resolved on the list. I'd be happy if you applied the patch.
          bodewig Stefan Bodewig added a comment -

          damjan: the patch doesn't apply cleanly anymore and if I fix it, your authorship information is lost. Do you think you could rebase it (or even better apply it yourself)

          bodewig Stefan Bodewig added a comment - damjan : the patch doesn't apply cleanly anymore and if I fix it, your authorship information is lost. Do you think you could rebase it (or even better apply it yourself)
          bodewig Stefan Bodewig added a comment -

          Current git master's 7z package is able to deal with SeekableByteChannel and doesn't require RandomAccessFile anymore. I plan to do the same to the zip package. The current code could be improved to use cached {{ByteBuffer}}s in order to reduce memory consumption.

          We may want to add an implementation of SeekableByteChannel wrapped around a byte[]}.

          bodewig Stefan Bodewig added a comment - Current git master's 7z package is able to deal with SeekableByteChannel and doesn't require RandomAccessFile anymore. I plan to do the same to the zip package. The current code could be improved to use cached {{ByteBuffer}}s in order to reduce memory consumption. We may want to add an implementation of SeekableByteChannel wrapped around a byte[] }.
          bodewig Stefan Bodewig added a comment -

          zip should now work as well, I'll close this ticket once I've updated the docs and at least a little bit optimized 7z

          bodewig Stefan Bodewig added a comment - zip should now work as well, I'll close this ticket once I've updated the docs and at least a little bit optimized 7z
          typek_pb Peter Butkovic added a comment -

          bodewig this is great news! Any time-wise plans on release holding this feature you could share with us?

          typek_pb Peter Butkovic added a comment - bodewig this is great news! Any time-wise plans on release holding this feature you could share with us?
          bodewig Stefan Bodewig added a comment -

          The past months have been very quiet for compress so this is the only bigger change since 1.12. There are a few other open bugs where we are waiting for feedback. I can't promise anything but wouldn't expect the next release to be more than a few weeks away.

          bodewig Stefan Bodewig added a comment - The past months have been very quiet for compress so this is the only bigger change since 1.12. There are a few other open bugs where we are waiting for feedback. I can't promise anything but wouldn't expect the next release to be more than a few weeks away.

          Added tests for SeekableInMemoryByteChannel and minor fixes.

          machiev Maciej Nowakowski added a comment - Added tests for SeekableInMemoryByteChannel and minor fixes.
          bodewig Stefan Bodewig added a comment -

          applied a modified version of your patch as git commit 72b1789 - thanks!

          bodewig Stefan Bodewig added a comment - applied a modified version of your patch as git commit 72b1789 - thanks!

          People

            Unassigned Unassigned
            bjkail Brett Kail
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: