Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 1.2
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Linux (CentOS) server; all client platforms and browsers

      Description

      I have been struggling for several weeks trying to track down the root cause
      of a sporadic file corruption problem using File Upload. I'm really stumped
      at this point and welcome any suggestions as to avenues of debugging
      pursuit.

      Here's overview of the problem:

      I have eight Linux (CentOS) servers all running the same web application--a
      set of Java servlets using Resin as a servlet runner under Apache. All
      servers were configured using the same script that installs jars,
      config files, etc.

      On three of the servers, File Upload works reliably. On five of the
      servers, File Upload usually (but not always) leaves me with a corrupted file.

      Corrupted files are always the correct length but contain a relatively small
      percentage of scrambled bytes. I looked for obvious patterns like newlines
      being altered or high-bit bytes being converted to or from UTF-8, but there
      is no obvious (to me, anyway) pattern in the failure.

      I have also tested with IE, FireFox, and Safari on both Windows and MacOS.
      The issue appears to be independent of client browser and OS.

      Looking at Java system properties, I notice that while the classpath and
      bootclasspath have the same jars lists on all servers, they are not listed
      in the same order (probably listed in directory order as the paths are
      constructed by a script that inspects lib directories.)

      Is anyone aware of classpath order dependencies that could break File
      Upload?

      Can anyone offer any suggestions about what might be breaking File Upload?
      Or what other questions I should be asking? At the moment, I'm feeling
      rather stumped.

        Activity

        Hide
        F. Andy Seidl added a comment -

        While the symptoms were real, the cause was outside (below) FileUpload. Specifically, within MultipartStream.makeAvailable() , the following function was violating its contract:

        int bytesRead = input.read(buffer, tail, bufSize- tail);

        Under certain not-understood but repeatable circumstances, the call to read() was acting as if tail==0 when it actually was non-zero (i.e., tail==pad). When this occurred, new data was written to the front of the buffer instead of tail bytes into the buffer. The result was tail bytes of data being dropped from the buffer, the new data being shifted to the front of the buffer, and tail bytes of garbage being re-used at the end of the buffer--in short, a corrupted file that retained its correct size (except for the even rarer occurrences when the lost bytes contained a boundary separator.)

        Ultimately the problem was in the implementation of the input stream class, which in this case, was com.caucho.server.connection.ServletInputStreamImpl, which is supplied by Resin.

        I did not determine under what circumstances this stream implementation was failing, but noting that read() was always returning 1460 bytes (a TCP packet size), even though the buffer was 4K, suggested that this was not a buffered implementation. I wrapped the raw input stream in a BufferedInputStream (by overriding getInputStream in ServletRequestContext) and the problems disappeared.

        Show
        F. Andy Seidl added a comment - While the symptoms were real, the cause was outside (below) FileUpload. Specifically, within MultipartStream.makeAvailable() , the following function was violating its contract: int bytesRead = input.read(buffer, tail, bufSize- tail); Under certain not-understood but repeatable circumstances, the call to read() was acting as if tail==0 when it actually was non-zero (i.e., tail==pad). When this occurred, new data was written to the front of the buffer instead of tail bytes into the buffer. The result was tail bytes of data being dropped from the buffer, the new data being shifted to the front of the buffer, and tail bytes of garbage being re-used at the end of the buffer--in short, a corrupted file that retained its correct size (except for the even rarer occurrences when the lost bytes contained a boundary separator.) Ultimately the problem was in the implementation of the input stream class, which in this case, was com.caucho.server.connection.ServletInputStreamImpl, which is supplied by Resin. I did not determine under what circumstances this stream implementation was failing, but noting that read() was always returning 1460 bytes (a TCP packet size), even though the buffer was 4K, suggested that this was not a buffered implementation. I wrapped the raw input stream in a BufferedInputStream (by overriding getInputStream in ServletRequestContext) and the problems disappeared.
        Hide
        F. Andy Seidl added a comment -

        Very interesting results. Does this trigger any ideas?

        My code writes uploaded files do disk by direclty copying the FileItem's
        input stream to the final file location. As an experiment to rule out a
        problem in the this copy operation, I createed a small (1112 byte) .jpg
        image file named fu-0.jpg. Then I created nine exact copies of the
        file: fu-1.jpg, fu-2.jpg, ..., fu-9.jpg.

        I FTP-ed fu-0.jpg to the server and I uploaded files 1-9 using
        FileUpload in a single upload request with nine files attached in
        numeric order. All the files uploaded without error except files 4 and
        5.

        Here is where it gets interesting...

        Look at the sizes of the uploaded files:

        fu-0.jpg: 1112
        fu-1.jpg: 1112
        fu-2.jpg: 1112
        fu-3.jpg: 1112
        fu-4.jpg: 1155 <=== wrong size
        fu-5.jpg: 1069 <=== wrong size
        fu-6.jpg: 1112
        fu-7.jpg: 1112
        fu-8.jpg: 1112
        fu-9.jpg: 1112

        BUT... file 4 and 5 add up to 2224 bytes, which is exactly two times the
        correct size of 1112 bytes. Thus, a worthy hypothesis is that the code
        that parsed the raw servlet request stream somehow missed the the
        boundary between the 4th and 5th file items.

        To rule out an alternate hypothesis that this was a random glitch
        unrelated to recognizing item boundaries, I repeated the test with ten
        new files, fu-a.jpg, fu-b.jpg, ..., fu-i.jpg. But this time, instead
        FileUpload-ing nine files at once, I FileUploaded each file separately.
        In this case, all ten files uploaded without error.

        Show
        F. Andy Seidl added a comment - Very interesting results. Does this trigger any ideas? My code writes uploaded files do disk by direclty copying the FileItem's input stream to the final file location. As an experiment to rule out a problem in the this copy operation, I createed a small (1112 byte) .jpg image file named fu-0.jpg. Then I created nine exact copies of the file: fu-1.jpg, fu-2.jpg, ..., fu-9.jpg. I FTP-ed fu-0.jpg to the server and I uploaded files 1-9 using FileUpload in a single upload request with nine files attached in numeric order. All the files uploaded without error except files 4 and 5. Here is where it gets interesting... Look at the sizes of the uploaded files: fu-0.jpg: 1112 fu-1.jpg: 1112 fu-2.jpg: 1112 fu-3.jpg: 1112 fu-4.jpg: 1155 <=== wrong size fu-5.jpg: 1069 <=== wrong size fu-6.jpg: 1112 fu-7.jpg: 1112 fu-8.jpg: 1112 fu-9.jpg: 1112 BUT... file 4 and 5 add up to 2224 bytes, which is exactly two times the correct size of 1112 bytes. Thus, a worthy hypothesis is that the code that parsed the raw servlet request stream somehow missed the the boundary between the 4th and 5th file items. To rule out an alternate hypothesis that this was a random glitch unrelated to recognizing item boundaries, I repeated the test with ten new files, fu-a.jpg, fu-b.jpg, ..., fu-i.jpg. But this time, instead FileUpload-ing nine files at once, I FileUploaded each file separately. In this case, all ten files uploaded without error.
        Hide
        F. Andy Seidl added a comment -

        Hi All,

        I'm back to debugging this FileUpload corruption issue. I've just installed commons-fileupload-1.2.1 and commons-io-1.4 and ran a simple test...

        I started with a local .jpg image, fileupload-test.jpg, and created two copies of that file: fileupload-test-copy.jpg and fileupload-test-copy-2.jpg. I uploaded the first two using FileUpload here:

        http://faseidl.com/docs/fileupload-test.jpg
        http://faseidl.com/docs/fileupload-test-copy.jpg

        I FTP-ed the third file here:

        http://faseidl.com/docs/fileupload-test-copy-2.jpg

        As you can see, the FileUpload-ed files are corrupted while the FTP-ed file is not.

        This situation occurs on five different servers while things work great on six others. All hypotheses welcome.

        Show
        F. Andy Seidl added a comment - Hi All, I'm back to debugging this FileUpload corruption issue. I've just installed commons-fileupload-1.2.1 and commons-io-1.4 and ran a simple test... I started with a local .jpg image, fileupload-test.jpg, and created two copies of that file: fileupload-test-copy.jpg and fileupload-test-copy-2.jpg. I uploaded the first two using FileUpload here: http://faseidl.com/docs/fileupload-test.jpg http://faseidl.com/docs/fileupload-test-copy.jpg I FTP-ed the third file here: http://faseidl.com/docs/fileupload-test-copy-2.jpg As you can see, the FileUpload-ed files are corrupted while the FTP-ed file is not. This situation occurs on five different servers while things work great on six others. All hypotheses welcome.
        Hide
        F. Andy Seidl added a comment -

        Hi Jochen (and others):

        I recently re-tried FileUpload on the affected Linux servers after upgrading to the latest release version of commons-io, but the issues remain. I realize that Jochen does not believe these results, but I remain stumped as to why the same code runs on most of my servers but corrupts files on others. It must be some type of environmental issue, but like what? This is particularly nasty for me to debug (lacking any guidance or hypotheses) because I only see failures on a small number of production machines. I have not been able to reproduce these failures on a development system. As such, it is a very slow debugging cycle to deploy debugging code and do bug hunting on a production server.

        Any suggestions as to what might create the type of corruptions I am seeing (and I really am seeing them) would be greatly appreciated.

        Thank you,
        – fas

        Show
        F. Andy Seidl added a comment - Hi Jochen (and others): I recently re-tried FileUpload on the affected Linux servers after upgrading to the latest release version of commons-io, but the issues remain. I realize that Jochen does not believe these results, but I remain stumped as to why the same code runs on most of my servers but corrupts files on others. It must be some type of environmental issue, but like what? This is particularly nasty for me to debug (lacking any guidance or hypotheses) because I only see failures on a small number of production machines. I have not been able to reproduce these failures on a development system. As such, it is a very slow debugging cycle to deploy debugging code and do bug hunting on a production server. Any suggestions as to what might create the type of corruptions I am seeing (and I really am seeing them) would be greatly appreciated. Thank you, – fas
        Hide
        F. Andy Seidl added a comment -

        Hi Jochen:

        >>I studied the mail from F. Andy Seidl you are quoting and I must admit that I simply don't believe it. <<
        I can understand that, but I assure you, this is the case. (If you are interested, I can arrange to provide you with credentials to a site you can experience this first-hand.)

        I have the same application deployed on eight Linux servers and one Widows server. File uploading works flawlessly and reliably on three of the Linux servers and the Windows server. On five of the Linux servers are the symptoms I documented above.

        I will try using the current snapshot as you recommend. But in the mean time, if anyone can think of anything that might cause this server-dependent failure, I'm eager for suggestions. I've been scratching my head over this one for quite some time.

        Thanks.

        Show
        F. Andy Seidl added a comment - Hi Jochen: >>I studied the mail from F. Andy Seidl you are quoting and I must admit that I simply don't believe it. << I can understand that, but I assure you, this is the case. (If you are interested, I can arrange to provide you with credentials to a site you can experience this first-hand.) I have the same application deployed on eight Linux servers and one Widows server. File uploading works flawlessly and reliably on three of the Linux servers and the Windows server. On five of the Linux servers are the symptoms I documented above. I will try using the current snapshot as you recommend. But in the mean time, if anyone can think of anything that might cause this server-dependent failure, I'm eager for suggestions. I've been scratching my head over this one for quite some time. Thanks.
        Hide
        Jochen Wiedmann added a comment -

        I studied the mail from F. Andy Seidl you are quoting and I must admit that I simply don't believe it. If the problems were as extreme as he states (no successful upload at all), then commons fileupload would be completely useless. Which it is not, because quite a lot of people use it in the wild.

        Show
        Jochen Wiedmann added a comment - I studied the mail from F. Andy Seidl you are quoting and I must admit that I simply don't believe it. If the problems were as extreme as he states (no successful upload at all), then commons fileupload would be completely useless. Which it is not, because quite a lot of people use it in the wild.
        Hide
        Jochen Wiedmann added a comment -

        The only confirmed problem with the current release is FILEUPLOAD-135. There is also FILEUPLOAD-144, which I suspect to be a duplicate. Please try to use a current snapshot, as available from

        http://people.apache.org/repo/m2-snapshot-repository/org/apache/commons/fileupload/commons-fileupload/1.3-SNAPSHOT/

        Show
        Jochen Wiedmann added a comment - The only confirmed problem with the current release is FILEUPLOAD-135 . There is also FILEUPLOAD-144 , which I suspect to be a duplicate. Please try to use a current snapshot, as available from http://people.apache.org/repo/m2-snapshot-repository/org/apache/commons/fileupload/commons-fileupload/1.3-SNAPSHOT/
        Hide
        F. Andy Seidl added a comment -

        I had placed this question to the mail list back in August. Here is one message that includes some additional details as well as some test results.

        ----Original Message----
        From: F. Andy Seidl faseidl@myst-technology.com
        Sent: Wednesday, August 22, 2007 11:25 AM
        To: user@commons.apache.org
        Subject: FW: File Upload Data Corruption Problem

        >From: Tahir Akhtar
        >
        >Do the files that are corrupted ends up same at each upload or there is
        >some randomness in that too?

        [fas] There is randomness in how each file is corrupted. See test results
        below.

        >Could it be something outside the file upload component, like disk
        >problem or another process corrupting the file after it has been written
        >on the server?

        [fas] This seems unlikely. This problem occurs on four different servers,
        none of which are having any other signs of trouble. As part of the tests
        below, I uploaded a file and then (from a command line session) immediately
        renamed the file.

        [snip]

        TEST RESULTS
        ------------

        I uploaded a file, pincney_map.pdf, four times. Immediately after each
        upload, I renamed the file to 1.pdf, 2.pdf, etc.

        Diff reports that each of the uploaded files differ from all the others.

        I then downloaded the four uploaded pdf files (using WinSCP, which has
        proven reliable for binary file transfer) to a Windows machine in order to
        use the "comp" utility to determine where the first difference was between
        each file using this simple script:

        @echo off

        comp pinckney_map.pdf 1.pdf
        comp pinckney_map.pdf 2.pdf
        comp pinckney_map.pdf 3.pdf
        comp pinckney_map.pdf 4.pdf

        comp 1.pdf 2.pdf
        comp 1.pdf 3.pdf
        comp 1.pdf 4.pdf
        comp 2.pdf 3.pdf
        comp 2.pdf 4.pdf
        comp 3.pdf 4.pd

        The output of that script follows:

        Comparing pinckney_map.pdf and 1.pdf...
        Compare error at OFFSET 1946
        file1 = 1
        file2 = 54
        Compare error at OFFSET 1947
        file1 = 17
        file2 = 61
        Compare error at OFFSET 1948
        file1 = 4
        file2 = D3
        Compare error at OFFSET 1949
        file1 = 40
        file2 = 10
        Compare error at OFFSET 194A
        file1 = 58
        file2 = 54
        Compare error at OFFSET 194B
        file1 = AA
        file2 = 55
        Compare error at OFFSET 194C
        file1 = EB
        file2 = BD
        Compare error at OFFSET 194D
        file1 = 3
        file2 = 7D
        Compare error at OFFSET 194E
        file1 = B
        file2 = 35
        Compare error at OFFSET 194F
        file1 = C3
        file2 = 33
        10 mismatches - ending compare

        Comparing pinckney_map.pdf and 2.pdf...
        Compare error at OFFSET 194C
        file1 = EB
        file2 = 55
        Compare error at OFFSET 194D
        file1 = 3
        file2 = BD
        Compare error at OFFSET 194E
        file1 = B
        file2 = 7D
        Compare error at OFFSET 194F
        file1 = C3
        file2 = 35
        Compare error at OFFSET 1950
        file1 = 5C
        file2 = 33
        Compare error at OFFSET 1951
        file1 = 15
        file2 = 78
        Compare error at OFFSET 1952
        file1 = B4
        file2 = 93
        Compare error at OFFSET 1953
        file1 = 77
        file2 = 60
        Compare error at OFFSET 1954
        file1 = F
        file2 = 16
        Compare error at OFFSET 1955
        file1 = 3
        file2 = 99
        10 mismatches - ending compare

        Comparing pinckney_map.pdf and 3.pdf...
        Compare error at OFFSET 1940
        file1 = 7D
        file2 = 20
        Compare error at OFFSET 1941
        file1 = D9
        file2 = 35
        Compare error at OFFSET 1942
        file1 = 38
        file2 = 59
        Compare error at OFFSET 1943
        file1 = 6D
        file2 = 61
        Compare error at OFFSET 1944
        file1 = 78
        file2 = 2C
        Compare error at OFFSET 1945
        file1 = EA
        file2 = 54
        Compare error at OFFSET 1946
        file1 = 1
        file2 = 61
        Compare error at OFFSET 1947
        file1 = 17
        file2 = D3
        Compare error at OFFSET 1948
        file1 = 4
        file2 = 10
        Compare error at OFFSET 1949
        file1 = 40
        file2 = 54
        10 mismatches - ending compare

        Comparing pinckney_map.pdf and 4.pdf...
        Compare error at OFFSET 1946
        file1 = 1
        file2 = 54
        Compare error at OFFSET 1947
        file1 = 17
        file2 = 61
        Compare error at OFFSET 1948
        file1 = 4
        file2 = D3
        Compare error at OFFSET 1949
        file1 = 40
        file2 = 10
        Compare error at OFFSET 194A
        file1 = 58
        file2 = 54
        Compare error at OFFSET 194B
        file1 = AA
        file2 = 55
        Compare error at OFFSET 194C
        file1 = EB
        file2 = BD
        Compare error at OFFSET 194D
        file1 = 3
        file2 = 7D
        Compare error at OFFSET 194E
        file1 = B
        file2 = 35
        Compare error at OFFSET 194F
        file1 = C3
        file2 = 33
        10 mismatches - ending compare

        Comparing 1.pdf and 2.pdf...
        Compare error at OFFSET 1946
        file1 = 54
        file2 = 1
        Compare error at OFFSET 1947
        file1 = 61
        file2 = 17
        Compare error at OFFSET 1948
        file1 = D3
        file2 = 4
        Compare error at OFFSET 1949
        file1 = 10
        file2 = 40
        Compare error at OFFSET 194A
        file1 = 54
        file2 = 58
        Compare error at OFFSET 194B
        file1 = 55
        file2 = AA
        Compare error at OFFSET 194C
        file1 = BD
        file2 = 55
        Compare error at OFFSET 194D
        file1 = 7D
        file2 = BD
        Compare error at OFFSET 194E
        file1 = 35
        file2 = 7D
        Compare error at OFFSET 194F
        file1 = 33
        file2 = 35
        10 mismatches - ending compare

        Comparing 1.pdf and 3.pdf...
        Compare error at OFFSET 1940
        file1 = 7D
        file2 = 20
        Compare error at OFFSET 1941
        file1 = D9
        file2 = 35
        Compare error at OFFSET 1942
        file1 = 38
        file2 = 59
        Compare error at OFFSET 1943
        file1 = 6D
        file2 = 61
        Compare error at OFFSET 1944
        file1 = 78
        file2 = 2C
        Compare error at OFFSET 1945
        file1 = EA
        file2 = 54
        Compare error at OFFSET 1946
        file1 = 54
        file2 = 61
        Compare error at OFFSET 1947
        file1 = 61
        file2 = D3
        Compare error at OFFSET 1948
        file1 = D3
        file2 = 10
        Compare error at OFFSET 1949
        file1 = 10
        file2 = 54
        10 mismatches - ending compare

        Comparing 1.pdf and 4.pdf...
        Compare error at OFFSET 16CA6
        file1 = 25
        file2 = C3
        Compare error at OFFSET 16CA7
        file1 = 44
        file2 = D1
        Compare error at OFFSET 16CA8
        file1 = B5
        file2 = 8E
        Compare error at OFFSET 16CA9
        file1 = B6
        file2 = 20
        Compare error at OFFSET 16CAA
        file1 = 9D
        file2 = F2
        Compare error at OFFSET 16CAB
        file1 = C5
        file2 = 6B
        Compare error at OFFSET 16CAC
        file1 = BA
        file2 = 63
        Compare error at OFFSET 16CAD
        file1 = C5
        file2 = ED
        Compare error at OFFSET 16CAE
        file1 = CD
        file2 = 98
        Compare error at OFFSET 16CAF
        file1 = EE
        file2 = 12
        10 mismatches - ending compare

        Comparing 2.pdf and 3.pdf...
        Compare error at OFFSET 1940
        file1 = 7D
        file2 = 20
        Compare error at OFFSET 1941
        file1 = D9
        file2 = 35
        Compare error at OFFSET 1942
        file1 = 38
        file2 = 59
        Compare error at OFFSET 1943
        file1 = 6D
        file2 = 61
        Compare error at OFFSET 1944
        file1 = 78
        file2 = 2C
        Compare error at OFFSET 1945
        file1 = EA
        file2 = 54
        Compare error at OFFSET 1946
        file1 = 1
        file2 = 61
        Compare error at OFFSET 1947
        file1 = 17
        file2 = D3
        Compare error at OFFSET 1948
        file1 = 4
        file2 = 10
        Compare error at OFFSET 1949
        file1 = 40
        file2 = 54
        10 mismatches - ending compare

        Comparing 2.pdf and 4.pdf...
        Compare error at OFFSET 1946
        file1 = 1
        file2 = 54
        Compare error at OFFSET 1947
        file1 = 17
        file2 = 61
        Compare error at OFFSET 1948
        file1 = 4
        file2 = D3
        Compare error at OFFSET 1949
        file1 = 40
        file2 = 10
        Compare error at OFFSET 194A
        file1 = 58
        file2 = 54
        Compare error at OFFSET 194B
        file1 = AA
        file2 = 55
        Compare error at OFFSET 194C
        file1 = 55
        file2 = BD
        Compare error at OFFSET 194D
        file1 = BD
        file2 = 7D
        Compare error at OFFSET 194E
        file1 = 7D
        file2 = 35
        Compare error at OFFSET 194F
        file1 = 35
        file2 = 33
        10 mismatches - ending compare

        Comparing 3.pdf and 4.pdf...
        Compare error at OFFSET 1940
        file1 = 20
        file2 = 7D
        Compare error at OFFSET 1941
        file1 = 35
        file2 = D9
        Compare error at OFFSET 1942
        file1 = 59
        file2 = 38
        Compare error at OFFSET 1943
        file1 = 61
        file2 = 6D
        Compare error at OFFSET 1944
        file1 = 2C
        file2 = 78
        Compare error at OFFSET 1945
        file1 = 54
        file2 = EA
        Compare error at OFFSET 1946
        file1 = 61
        file2 = 54
        Compare error at OFFSET 1947
        file1 = D3
        file2 = 61
        Compare error at OFFSET 1948
        file1 = 10
        file2 = D3
        Compare error at OFFSET 1949
        file1 = 54
        file2 = 10
        10 mismatches - ending compare

        ---------------------------------------------------------------------
        To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
        For additional commands, e-mail: user-help@commons.apache.org

        Show
        F. Andy Seidl added a comment - I had placed this question to the mail list back in August. Here is one message that includes some additional details as well as some test results. ---- Original Message ---- From: F. Andy Seidl faseidl@myst-technology.com Sent: Wednesday, August 22, 2007 11:25 AM To: user@commons.apache.org Subject: FW: File Upload Data Corruption Problem >From: Tahir Akhtar > >Do the files that are corrupted ends up same at each upload or there is >some randomness in that too? [fas] There is randomness in how each file is corrupted. See test results below. >Could it be something outside the file upload component, like disk >problem or another process corrupting the file after it has been written >on the server? [fas] This seems unlikely. This problem occurs on four different servers, none of which are having any other signs of trouble. As part of the tests below, I uploaded a file and then (from a command line session) immediately renamed the file. [snip] TEST RESULTS ------------ I uploaded a file, pincney_map.pdf, four times. Immediately after each upload, I renamed the file to 1.pdf, 2.pdf, etc. Diff reports that each of the uploaded files differ from all the others. I then downloaded the four uploaded pdf files (using WinSCP, which has proven reliable for binary file transfer) to a Windows machine in order to use the "comp" utility to determine where the first difference was between each file using this simple script: @echo off comp pinckney_map.pdf 1.pdf comp pinckney_map.pdf 2.pdf comp pinckney_map.pdf 3.pdf comp pinckney_map.pdf 4.pdf comp 1.pdf 2.pdf comp 1.pdf 3.pdf comp 1.pdf 4.pdf comp 2.pdf 3.pdf comp 2.pdf 4.pdf comp 3.pdf 4.pd The output of that script follows: Comparing pinckney_map.pdf and 1.pdf... Compare error at OFFSET 1946 file1 = 1 file2 = 54 Compare error at OFFSET 1947 file1 = 17 file2 = 61 Compare error at OFFSET 1948 file1 = 4 file2 = D3 Compare error at OFFSET 1949 file1 = 40 file2 = 10 Compare error at OFFSET 194A file1 = 58 file2 = 54 Compare error at OFFSET 194B file1 = AA file2 = 55 Compare error at OFFSET 194C file1 = EB file2 = BD Compare error at OFFSET 194D file1 = 3 file2 = 7D Compare error at OFFSET 194E file1 = B file2 = 35 Compare error at OFFSET 194F file1 = C3 file2 = 33 10 mismatches - ending compare Comparing pinckney_map.pdf and 2.pdf... Compare error at OFFSET 194C file1 = EB file2 = 55 Compare error at OFFSET 194D file1 = 3 file2 = BD Compare error at OFFSET 194E file1 = B file2 = 7D Compare error at OFFSET 194F file1 = C3 file2 = 35 Compare error at OFFSET 1950 file1 = 5C file2 = 33 Compare error at OFFSET 1951 file1 = 15 file2 = 78 Compare error at OFFSET 1952 file1 = B4 file2 = 93 Compare error at OFFSET 1953 file1 = 77 file2 = 60 Compare error at OFFSET 1954 file1 = F file2 = 16 Compare error at OFFSET 1955 file1 = 3 file2 = 99 10 mismatches - ending compare Comparing pinckney_map.pdf and 3.pdf... Compare error at OFFSET 1940 file1 = 7D file2 = 20 Compare error at OFFSET 1941 file1 = D9 file2 = 35 Compare error at OFFSET 1942 file1 = 38 file2 = 59 Compare error at OFFSET 1943 file1 = 6D file2 = 61 Compare error at OFFSET 1944 file1 = 78 file2 = 2C Compare error at OFFSET 1945 file1 = EA file2 = 54 Compare error at OFFSET 1946 file1 = 1 file2 = 61 Compare error at OFFSET 1947 file1 = 17 file2 = D3 Compare error at OFFSET 1948 file1 = 4 file2 = 10 Compare error at OFFSET 1949 file1 = 40 file2 = 54 10 mismatches - ending compare Comparing pinckney_map.pdf and 4.pdf... Compare error at OFFSET 1946 file1 = 1 file2 = 54 Compare error at OFFSET 1947 file1 = 17 file2 = 61 Compare error at OFFSET 1948 file1 = 4 file2 = D3 Compare error at OFFSET 1949 file1 = 40 file2 = 10 Compare error at OFFSET 194A file1 = 58 file2 = 54 Compare error at OFFSET 194B file1 = AA file2 = 55 Compare error at OFFSET 194C file1 = EB file2 = BD Compare error at OFFSET 194D file1 = 3 file2 = 7D Compare error at OFFSET 194E file1 = B file2 = 35 Compare error at OFFSET 194F file1 = C3 file2 = 33 10 mismatches - ending compare Comparing 1.pdf and 2.pdf... Compare error at OFFSET 1946 file1 = 54 file2 = 1 Compare error at OFFSET 1947 file1 = 61 file2 = 17 Compare error at OFFSET 1948 file1 = D3 file2 = 4 Compare error at OFFSET 1949 file1 = 10 file2 = 40 Compare error at OFFSET 194A file1 = 54 file2 = 58 Compare error at OFFSET 194B file1 = 55 file2 = AA Compare error at OFFSET 194C file1 = BD file2 = 55 Compare error at OFFSET 194D file1 = 7D file2 = BD Compare error at OFFSET 194E file1 = 35 file2 = 7D Compare error at OFFSET 194F file1 = 33 file2 = 35 10 mismatches - ending compare Comparing 1.pdf and 3.pdf... Compare error at OFFSET 1940 file1 = 7D file2 = 20 Compare error at OFFSET 1941 file1 = D9 file2 = 35 Compare error at OFFSET 1942 file1 = 38 file2 = 59 Compare error at OFFSET 1943 file1 = 6D file2 = 61 Compare error at OFFSET 1944 file1 = 78 file2 = 2C Compare error at OFFSET 1945 file1 = EA file2 = 54 Compare error at OFFSET 1946 file1 = 54 file2 = 61 Compare error at OFFSET 1947 file1 = 61 file2 = D3 Compare error at OFFSET 1948 file1 = D3 file2 = 10 Compare error at OFFSET 1949 file1 = 10 file2 = 54 10 mismatches - ending compare Comparing 1.pdf and 4.pdf... Compare error at OFFSET 16CA6 file1 = 25 file2 = C3 Compare error at OFFSET 16CA7 file1 = 44 file2 = D1 Compare error at OFFSET 16CA8 file1 = B5 file2 = 8E Compare error at OFFSET 16CA9 file1 = B6 file2 = 20 Compare error at OFFSET 16CAA file1 = 9D file2 = F2 Compare error at OFFSET 16CAB file1 = C5 file2 = 6B Compare error at OFFSET 16CAC file1 = BA file2 = 63 Compare error at OFFSET 16CAD file1 = C5 file2 = ED Compare error at OFFSET 16CAE file1 = CD file2 = 98 Compare error at OFFSET 16CAF file1 = EE file2 = 12 10 mismatches - ending compare Comparing 2.pdf and 3.pdf... Compare error at OFFSET 1940 file1 = 7D file2 = 20 Compare error at OFFSET 1941 file1 = D9 file2 = 35 Compare error at OFFSET 1942 file1 = 38 file2 = 59 Compare error at OFFSET 1943 file1 = 6D file2 = 61 Compare error at OFFSET 1944 file1 = 78 file2 = 2C Compare error at OFFSET 1945 file1 = EA file2 = 54 Compare error at OFFSET 1946 file1 = 1 file2 = 61 Compare error at OFFSET 1947 file1 = 17 file2 = D3 Compare error at OFFSET 1948 file1 = 4 file2 = 10 Compare error at OFFSET 1949 file1 = 40 file2 = 54 10 mismatches - ending compare Comparing 2.pdf and 4.pdf... Compare error at OFFSET 1946 file1 = 1 file2 = 54 Compare error at OFFSET 1947 file1 = 17 file2 = 61 Compare error at OFFSET 1948 file1 = 4 file2 = D3 Compare error at OFFSET 1949 file1 = 40 file2 = 10 Compare error at OFFSET 194A file1 = 58 file2 = 54 Compare error at OFFSET 194B file1 = AA file2 = 55 Compare error at OFFSET 194C file1 = 55 file2 = BD Compare error at OFFSET 194D file1 = BD file2 = 7D Compare error at OFFSET 194E file1 = 7D file2 = 35 Compare error at OFFSET 194F file1 = 35 file2 = 33 10 mismatches - ending compare Comparing 3.pdf and 4.pdf... Compare error at OFFSET 1940 file1 = 20 file2 = 7D Compare error at OFFSET 1941 file1 = 35 file2 = D9 Compare error at OFFSET 1942 file1 = 59 file2 = 38 Compare error at OFFSET 1943 file1 = 61 file2 = 6D Compare error at OFFSET 1944 file1 = 2C file2 = 78 Compare error at OFFSET 1945 file1 = 54 file2 = EA Compare error at OFFSET 1946 file1 = 61 file2 = 54 Compare error at OFFSET 1947 file1 = D3 file2 = 61 Compare error at OFFSET 1948 file1 = 10 file2 = D3 Compare error at OFFSET 1949 file1 = 54 file2 = 10 10 mismatches - ending compare --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@commons.apache.org For additional commands, e-mail: user-help@commons.apache.org

          People

          • Assignee:
            Unassigned
            Reporter:
            F. Andy Seidl
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development