Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0
    • Fix Version/s: 1.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      Java 1.4.2 run on AIX and Linux

      Description

      The code below is called with an array of 4 file names. The cpio archive archive.cpio is created with no error messages, but when I then run the Unix command "cpio -ivct <archive.cpio" it reports the error "Can't read input" on the last file in the archive. If I run "cpio -ivcBmu <archive.cpio" the last file is incomplete, but the other files are extracted correctly. Same result in AIX and Linux.

      {{
      private void createArchive(String[] outFiles)
      throws FileNotFoundException, IOException, ArchiveException {
      short format = CpioArchiveOutputStream.FORMAT_OLD_ASCII;
      final OutputStream out = new FileOutputStream("archive.cpio");
      ArchiveOutputStream os = new CpioArchiveOutputStream(out, format);

      for (int j = 0; j < outFiles.length; j++)

      { System.out.println("Entry = " + outFiles[j]); File f = new File(outFiles[j]); CpioArchiveEntry entry = new CpioArchiveEntry(format); entry.setName(outFiles[j]); entry.setSize(f.length()); os.putArchiveEntry(entry); IOUtils.copy(new FileInputStream(outFiles[j]), os); os.closeArchiveEntry(); }

      os.finish();
      os.close();
      }
      }}

      1. archive.cpio
        4 kB
        Stefan Bodewig
      2. archive2.cpio
        4 kB
        Stefan Bodewig
      3. archive3.cpio
        4 kB
        Bill Maier
      4. archive4.cpio
        5 kB
        Bill Maier
      5. cpio.tar
        10 kB
        Bill Maier
      6. Creator.java
        1 kB
        Stefan Bodewig

        Activity

        Hide
        Stefan Bodewig added a comment -

        It may be that the result depends on the actual files being archived. I've turned your snippet into the trivial program (soon to be) attached and created an archive of four files from the current svn tree of commons compress (as of 2010-02-15) and cpio on my Linux box is happy with the result created with either the 1.0 version or currsnt svn trunk.

        Would it be possible for you to provide a set of files where creation fails?

        Show
        Stefan Bodewig added a comment - It may be that the result depends on the actual files being archived. I've turned your snippet into the trivial program (soon to be) attached and created an archive of four files from the current svn tree of commons compress (as of 2010-02-15) and cpio on my Linux box is happy with the result created with either the 1.0 version or currsnt svn trunk. Would it be possible for you to provide a set of files where creation fails?
        Hide
        Bill Maier added a comment -

        I've attached a .tar file containing 4 class files and a readme.txt description. Saw the same problem when tried to put the 4 class files into a cpio archive. I used your file Creator.java to run. I suspect now that the files are not the problem - might be a combination of the older Java version we use (1.4.2) and the operating system. The attached readme.txt file shows the result in AIX. I also tried on Linux, which gave the error "cpio: premature end of file" when running cpio -ivct on the resulting file.

        Show
        Bill Maier added a comment - I've attached a .tar file containing 4 class files and a readme.txt description. Saw the same problem when tried to put the 4 class files into a cpio archive. I used your file Creator.java to run. I suspect now that the files are not the problem - might be a combination of the older Java version we use (1.4.2) and the operating system. The attached readme.txt file shows the result in AIX. I also tried on Linux, which gave the error "cpio: premature end of file" when running cpio -ivct on the resulting file.
        Hide
        Stefan Bodewig added a comment -

        At least the VM seems to be part of the problem, yes. I've just tried your classes (thanks!) on my Linux box and the archive seems to be fine, unfortunately I don't have a 1.4 JVM for Linux around, will try later today on my work Windows box where I have an old Sun 1.4.2 around.

        In any case, could you please also attach the archive that has been created so I can do a byte by byte comparison with the one I have created on Linux which works? This may help diagnosing the problem even if I can't reproduce it myself.

        Show
        Stefan Bodewig added a comment - At least the VM seems to be part of the problem, yes. I've just tried your classes (thanks!) on my Linux box and the archive seems to be fine, unfortunately I don't have a 1.4 JVM for Linux around, will try later today on my work Windows box where I have an old Sun 1.4.2 around. In any case, could you please also attach the archive that has been created so I can do a byte by byte comparison with the one I have created on Linux which works? This may help diagnosing the problem even if I can't reproduce it myself.
        Hide
        Stefan Bodewig added a comment -

        CPIO archive created with Creator.java from Bill's class files which seems to work fine on my Linux machine for reference

        Show
        Stefan Bodewig added a comment - CPIO archive created with Creator.java from Bill's class files which seems to work fine on my Linux machine for reference
        Hide
        Stefan Bodewig added a comment -

        I get the exact same cpio file using JDK 1.4.2 (Sun's) on Windows as well.

        Show
        Stefan Bodewig added a comment - I get the exact same cpio file using JDK 1.4.2 (Sun's) on Windows as well.
        Hide
        Bill Maier added a comment -

        Archive created by Creator class. The system cpio command on AIX and Linux is unable to read this file.

        Show
        Bill Maier added a comment - Archive created by Creator class. The system cpio command on AIX and Linux is unable to read this file.
        Hide
        Bill Maier added a comment -

        I downloaded the archive you attached, and copied it to my AIX machine. I get the same error when the system command cpio tries to read the file:

        [npd1:/home/bmaier/test/java/creator] cpio -ivct <archive.cpio
        0 root 593 Dec 31 18:00:00 1969 ApplicationException.class
        0 root 857 Dec 31 18:00:00 1969 BData.class
        0 root 342 Dec 31 18:00:00 1969 BufException.class
        Can't read input
        [npd1:/home/bmaier/test/java/creator]

        Compared Stefan's archive with mine and they are byte-for-byte identical. I'm running SUSE on the Linux machine:

        bmaier@ftw-srv-arv2-02:~> cat /etc/*-release
        LSB_VERSION="core-2.0-noarch:core-3.0-noarch:core-2.0-ia32:core-3.0-ia32"
        SUSE Linux Enterprise Server 10 (i586)
        VERSION = 10
        PATCHLEVEL = 2

        Show
        Bill Maier added a comment - I downloaded the archive you attached, and copied it to my AIX machine. I get the same error when the system command cpio tries to read the file: [npd1:/home/bmaier/test/java/creator] cpio -ivct <archive.cpio 0 root 593 Dec 31 18:00:00 1969 ApplicationException.class 0 root 857 Dec 31 18:00:00 1969 BData.class 0 root 342 Dec 31 18:00:00 1969 BufException.class Can't read input [npd1:/home/bmaier/test/java/creator] Compared Stefan's archive with mine and they are byte-for-byte identical. I'm running SUSE on the Linux machine: bmaier@ftw-srv-arv2-02:~> cat /etc/*-release LSB_VERSION="core-2.0-noarch:core-3.0-noarch:core-2.0-ia32:core-3.0-ia32" SUSE Linux Enterprise Server 10 (i586) VERSION = 10 PATCHLEVEL = 2
        Hide
        Stefan Bodewig added a comment -

        OK, so the problem is the version of cpio not the version of Java or the files - that's even worse.

        In my case it has been Ubuntu 9.04 which provides GNU cpio 2.9 (that's what cpio --version says). I'll see whether I do have access to a Unix machine running a different flavor of cpio.

        Does it help (as a workaround) if you switch to one of the other formats the CPIO classes provide?

        Show
        Stefan Bodewig added a comment - OK, so the problem is the version of cpio not the version of Java or the files - that's even worse. In my case it has been Ubuntu 9.04 which provides GNU cpio 2.9 (that's what cpio --version says). I'll see whether I do have access to a Unix machine running a different flavor of cpio. Does it help (as a workaround) if you switch to one of the other formats the CPIO classes provide?
        Hide
        Stefan Bodewig added a comment -

        The other one I've found so far (Solaris) doesn't like the archive at all

        $ cpio -ivct < archive.cpio
        cpio: Not a cpio file, bad header.
        1 errors

        I guess I have some reading on the different cpio formats to do ...

        Show
        Stefan Bodewig added a comment - The other one I've found so far (Solaris) doesn't like the archive at all $ cpio -ivct < archive.cpio cpio: Not a cpio file, bad header. 1 errors I guess I have some reading on the different cpio formats to do ...
        Hide
        Bill Maier added a comment -

        The cpio version on our SUSE is GNU cpio 2.6, quite a bit earlier than Stefan's version. Cpio on AIX doesn't seem to have an equivalent way of getting the version number.

        I have tried other formats (although for our specific application we need the old ASCII format). It has the same trouble with all the formats I've tried, FORMAT_NEW and FORMAT_OLD_BINARY. Sometimes I get an "Out of phase" error, but I think this is when my flags don't match the archive format:

        [npd1:/home/bmaier/test/java/creator] cpio -ivct <archive.cpio

        Out of phase!
        cpio attempting to continue...
        Can't read input

        Show
        Bill Maier added a comment - The cpio version on our SUSE is GNU cpio 2.6, quite a bit earlier than Stefan's version. Cpio on AIX doesn't seem to have an equivalent way of getting the version number. I have tried other formats (although for our specific application we need the old ASCII format). It has the same trouble with all the formats I've tried, FORMAT_NEW and FORMAT_OLD_BINARY. Sometimes I get an "Out of phase" error, but I think this is when my flags don't match the archive format: [npd1:/home/bmaier/test/java/creator] cpio -ivct <archive.cpio Out of phase! cpio attempting to continue... Can't read input
        Hide
        Stefan Bodewig added a comment -

        So far I've found a few places where we seem to deviate a bit from http://people.freebsd.org/~kientzle/libarchive/man/cpio.5.txt or our own description in the javadocs of CpioArchiveEntry. Mainly:

        • namelen could be wrong in non.ASCII cases - not the case with the example files
        • dev/ino are always 0 but should be different for each entry
        • nlink is always 0 but should at least be 1 for files and 2 for directories

        The permissions and gid/uid are also 0 but that should be OK. Neither of these things would explain why cpio would be happy with the first three entries but doesn't like the forth one. Can you isolate the forth one as the one causing the problem? I.e. do you get valid archives when putting only the first three into it, or just the forth one?

        Back into hexl-mode trying to see whether there is anything fishy with the last entry.

        Show
        Stefan Bodewig added a comment - So far I've found a few places where we seem to deviate a bit from http://people.freebsd.org/~kientzle/libarchive/man/cpio.5.txt or our own description in the javadocs of CpioArchiveEntry. Mainly: namelen could be wrong in non.ASCII cases - not the case with the example files dev/ino are always 0 but should be different for each entry nlink is always 0 but should at least be 1 for files and 2 for directories The permissions and gid/uid are also 0 but that should be OK. Neither of these things would explain why cpio would be happy with the first three entries but doesn't like the forth one. Can you isolate the forth one as the one causing the problem? I.e. do you get valid archives when putting only the first three into it, or just the forth one? Back into hexl-mode trying to see whether there is anything fishy with the last entry.
        Hide
        Bill Maier added a comment -

        Still get "can't read input" error from cpio -ivct when only 3 files are in the archive.

        Show
        Bill Maier added a comment - Still get "can't read input" error from cpio -ivct when only 3 files are in the archive.
        Hide
        Stefan Bodewig added a comment -

        After explicitly setting inode, device and nlink Solaris at least sees the entries:

        -bash-3.00$ cpio -ivct < archive.cpio
        cpio: Impossible file type
        ---------- 1 root root 593 Feb 16 16:08 2010, ApplicationException.class
        cpio: Impossible file type
        ---------- 1 root root 857 Feb 16 16:08 2010, BData.class
        cpio: Impossible file type
        ---------- 1 root root 342 Feb 16 16:08 2010, BufException.class
        cpio: Impossible file type
        ---------- 1 root root 1274 Feb 16 16:08 2010, ByteArrayUtil.class

        7 blocks
        4 error(s)

        Show
        Stefan Bodewig added a comment - After explicitly setting inode, device and nlink Solaris at least sees the entries: -bash-3.00$ cpio -ivct < archive.cpio cpio: Impossible file type ---------- 1 root root 593 Feb 16 16:08 2010, ApplicationException.class cpio: Impossible file type ---------- 1 root root 857 Feb 16 16:08 2010, BData.class cpio: Impossible file type ---------- 1 root root 342 Feb 16 16:08 2010, BufException.class cpio: Impossible file type ---------- 1 root root 1274 Feb 16 16:08 2010, ByteArrayUtil.class 7 blocks 4 error(s)
        Hide
        Stefan Bodewig added a comment -

        and now with explicit file mode. Solaris is happy. What does your cpio say?

        Show
        Stefan Bodewig added a comment - and now with explicit file mode. Solaris is happy. What does your cpio say?
        Hide
        Bill Maier added a comment -

        Got same error on AIX with the attached archive, but on SUSE cpio now reports "cpio:premature end of file".

        Show
        Bill Maier added a comment - Got same error on AIX with the attached archive, but on SUSE cpio now reports "cpio:premature end of file".
        Hide
        Stefan Bodewig added a comment -

        OK, I created a cpio archive of your example files with GNU cpio. It looks as if cpio archives are supposed to write files in blocks of 512 bytes - actually the block size is a command line option for cpio.

        The attached archive has been created with Creator.java and svn revision 911031 of commons compress. Does it work for your cpio versions?

        Show
        Stefan Bodewig added a comment - OK, I created a cpio archive of your example files with GNU cpio. It looks as if cpio archives are supposed to write files in blocks of 512 bytes - actually the block size is a command line option for cpio. The attached archive has been created with Creator.java and svn revision 911031 of commons compress. Does it work for your cpio versions?
        Hide
        Bill Maier added a comment -

        That archive works on AIX using either

        cpio -ivct <archive.cpio

        for listing or

        cpio -ivcmu <archive.cpio

        for extraction. Apparently the block size is a factor here, so you've
        made some progress. Interestingly it still doesn't work using

        cpio -ivcBmu <archive.cpio

        where the B option is supposed to force 512-byte blocks (?!).

        Cpio still can't read the archive in SUSE. I tried all of these commands:

        cpio -ivct <archive.cpio
        cpio -ivctB <archive.cpio
        cpio -ivctC512 <archive.cpio
        cpio -ivctC5120 <archive.cpio
        cpio -ivctC 5120 <archive.cpio
        cpio -ivctC 512 <archive.cpio
        cpio -ivct --block-size=512 <archive.cpio
        cpio -ivct --block-size=5120 <archive.cpio

        It always reports "cpio: premature end of file".

        Show
        Bill Maier added a comment - That archive works on AIX using either cpio -ivct <archive.cpio for listing or cpio -ivcmu <archive.cpio for extraction. Apparently the block size is a factor here, so you've made some progress. Interestingly it still doesn't work using cpio -ivcBmu <archive.cpio where the B option is supposed to force 512-byte blocks (?!). Cpio still can't read the archive in SUSE. I tried all of these commands: cpio -ivct <archive.cpio cpio -ivctB <archive.cpio cpio -ivctC512 <archive.cpio cpio -ivctC5120 <archive.cpio cpio -ivctC 5120 <archive.cpio cpio -ivctC 512 <archive.cpio cpio -ivct --block-size=512 <archive.cpio cpio -ivct --block-size=5120 <archive.cpio It always reports "cpio: premature end of file".
        Hide
        Stefan Bodewig added a comment -

        This is both frustrating and fun in a strange way.

        -B sets the block size to 5120 while the archive is created with a size of 512, so it isn't supposed to work. With svn revision 911353 I've made the block size configurable.

        I've also introduced some logic inside the library so you don't have to set inode and device explicitly in order to get unique values. nlink and mode should now at least provide reasonable default values.

        archive.cpio has been created with Creator.java and commons-compress svn revision 911353, archive2.cpio has been created with GNU cpio 2.9 on Ubuntu Linux 9.04. They only differ in device and inode numbers and timestamps.

        Since the archive doesn't work for your Suse system, could you please create a cpio archive of your our files and upload it here as well?

        Show
        Stefan Bodewig added a comment - This is both frustrating and fun in a strange way. -B sets the block size to 5120 while the archive is created with a size of 512, so it isn't supposed to work. With svn revision 911353 I've made the block size configurable. I've also introduced some logic inside the library so you don't have to set inode and device explicitly in order to get unique values. nlink and mode should now at least provide reasonable default values. archive.cpio has been created with Creator.java and commons-compress svn revision 911353, archive2.cpio has been created with GNU cpio 2.9 on Ubuntu Linux 9.04. They only differ in device and inode numbers and timestamps. Since the archive doesn't work for your Suse system, could you please create a cpio archive of your our files and upload it here as well?
        Hide
        Bill Maier added a comment -

        archive3.cpio was made with this command:

        ls *class |cpio -ovc >archive3.cpio

        archive4.cpio was made with this command:

        ls *class |cpio -ovcB >archive4.cpio

        Both were made on SUSE Linux (SLES 10).

        Show
        Bill Maier added a comment - archive3.cpio was made with this command: ls *class |cpio -ovc >archive3.cpio archive4.cpio was made with this command: ls *class |cpio -ovcB >archive4.cpio Both were made on SUSE Linux (SLES 10).
        Hide
        Stefan Bodewig added a comment - - edited

        both archives use the "new ascii" format, so it looks as if -c didn't work.

        If I use my GNU cpio 2.9 and "cpio -cvt < archive?.cpio" I now get the "premature end of file".

        The alternative to -c is "-H odc" - does that work to list/extract archive.cpio?

        Show
        Stefan Bodewig added a comment - - edited both archives use the "new ascii" format, so it looks as if -c didn't work. If I use my GNU cpio 2.9 and "cpio -cvt < archive?.cpio" I now get the "premature end of file". The alternative to -c is "-H odc" - does that work to list/extract archive.cpio?
        Hide
        Bill Maier added a comment -

        These commands work on cpio.archive for SUSE to list and extract:

        cpio -ivt -H odc <archive.cpio
        cpio -iv -H odc <archive.cpio

        So SUSE cpio won't do old ASCII format apparently.

        Show
        Bill Maier added a comment - These commands work on cpio.archive for SUSE to list and extract: cpio -ivt -H odc <archive.cpio cpio -iv -H odc <archive.cpio So SUSE cpio won't do old ASCII format apparently.
        Hide
        Stefan Bodewig added a comment -

        I'm not sure whether you are saying that the commands with -D odc work for my cpio archives on Suse or not. If they do then Suse's cpio works for old ascii but the -c switch is broken.

        In any case it means compress now creates cpio archives that can be extracted on AIX and Solaris and with GNU cpio 2.9. I'll resolve this issue once I've documented the fix.

        Show
        Stefan Bodewig added a comment - I'm not sure whether you are saying that the commands with -D odc work for my cpio archives on Suse or not. If they do then Suse's cpio works for old ascii but the -c switch is broken. In any case it means compress now creates cpio archives that can be extracted on AIX and Solaris and with GNU cpio 2.9. I'll resolve this issue once I've documented the fix.
        Hide
        Stefan Bodewig added a comment -

        fixed and documented with svn revision 911427

        Show
        Stefan Bodewig added a comment - fixed and documented with svn revision 911427

          People

          • Assignee:
            Unassigned
            Reporter:
            Bill Maier
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development