Fop
  1. Fop
  2. FOP-1256

[PATCH] png graphics are expanded/uncompressed in pdf causing massive file size increase

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Resolution: Fixed
    • Affects Version/s: trunk
    • Fix Version/s: None
    • Component/s: image/unqualified
    • Labels:
      None
    • Environment:
      Operating System: other
      Platform: PC
    • External issue ID:
      40676

      Description

      graphics as png files (total files size about 2 Mb) are expanded to some bitmap?
      representation in pdf leading to a massive increase in file size (from 0.6 Mb
      using version 2 to 10 Mb using version 0.92).

      Is there some way of specifying that compressed graphics should be used in the pdf.

      (The graphics are actually math equations - fairly complicated with integrals
      etc, and some simple graphs of functions like normal distribution. About 30 of
      each, so quite a lot of work to change representation Is there some other
      workaround? vector format?)

      1. example_images_srgb.tar.gz
        925 kB
        Luis Bernardo
      2. test-srgb-rendering_intent.pdf
        931 kB
        Luis Bernardo
      3. example_images_iccp.tar.gz
        583 kB
        Luis Bernardo
      4. test-iccp.pdf
        583 kB
        Luis Bernardo
      5. fop-bug40676.patch
        11 kB
        Luis Bernardo
      6. example.tar.gz
        760 kB
        Luis Bernardo
      7. test.tar.gz
        101 kB
        Luis Bernardo
      8. fop-disallowmultiplefilters.patch
        5 kB
        Luis Bernardo
      9. fop-png-documentation.patch
        3 kB
        Luis Bernardo
      10. bug40676-findbugs.patch
        2 kB
        Luis Bernardo
      11. example.tar.gz
        760 kB
        Luis Bernardo
      12. test.tar.gz
        101 kB
        Luis Bernardo
      13. apachefop.patch
        51 kB
        Luis Bernardo
      14. test.pdf
        162 kB
        Luis Bernardo

        Activity

        Hide
        Jeremias Maerki added a comment -

        PNG images are currently decompressed, normalized to RGBA (32bit) and then
        recompressed for PDF. Of course, that's suboptimal.

        Yes, it's theoretically possible to embed the compressed PNG data directly in
        the PDF. This is something that is on my task list for the next three months as
        part of a general redesign of the image adapter package which is also
        responsible for the implicit conversion to RGBA of the image.

        Work-arounds for you: Yes, please try a vector format, preferably SVG because we
        can handle this natively. Your equations will look much nicer.

        Show
        Jeremias Maerki added a comment - PNG images are currently decompressed, normalized to RGBA (32bit) and then recompressed for PDF. Of course, that's suboptimal. Yes, it's theoretically possible to embed the compressed PNG data directly in the PDF. This is something that is on my task list for the next three months as part of a general redesign of the image adapter package which is also responsible for the implicit conversion to RGBA of the image. Work-arounds for you: Yes, please try a vector format, preferably SVG because we can handle this natively. Your equations will look much nicer.
        Hide
        Paul A. Bristow added a comment -

        Thanks for helpful comment and promises But meanwhile: Do you any
        recommendations on MathML to SVG conversion. A quick Google turns up some tools
        but some seem not to work well yet.

        We also have a problem with simple x y graphs to show simple functions as
        coloured curves, for example normal distribution 'bell' curve. Do you have
        suggestions for doing these (from tables of data as decimal digits).

        Thanks

        Paul

        Show
        Paul A. Bristow added a comment - Thanks for helpful comment and promises But meanwhile: Do you any recommendations on MathML to SVG conversion. A quick Google turns up some tools but some seem not to work well yet. We also have a problem with simple x y graphs to show simple functions as coloured curves, for example normal distribution 'bell' curve. Do you have suggestions for doing these (from tables of data as decimal digits). Thanks Paul
        Hide
        Jeremias Maerki added a comment -

        We use JEuclid.
        See the FO extension in examples/mathml

        Show
        Jeremias Maerki added a comment - We use JEuclid. See the FO extension in examples/mathml
        Hide
        Glenn Adams added a comment -

        resetting P2 open bugs to P3 pending further review

        Show
        Glenn Adams added a comment - resetting P2 open bugs to P3 pending further review
        Hide
        Glenn Adams added a comment -

        change status from ASSIGNED to NEW for consistency

        Show
        Glenn Adams added a comment - change status from ASSIGNED to NEW for consistency
        Hide
        Luis Bernardo added a comment -

        Attachment test.pdf has been added with description: output PDF of test case after fix

        Show
        Luis Bernardo added a comment - Attachment test.pdf has been added with description: output PDF of test case after fix
        Hide
        Luis Bernardo added a comment -

        sorry, above attachment, 28771, is for FOP-1259. stupid bugzilla...

        Show
        Luis Bernardo added a comment - sorry, above attachment, 28771, is for FOP-1259 . stupid bugzilla...
        Hide
        Luis Bernardo added a comment -

        this patch includes new image handlers for PNG for PDF and PS output. these image handlers can handle the raw IDAT chunk and lead to smaller files.

        to use these handlers the ImageLoaderRawPNG needs to be enabled in the fop.xconf file.

        Show
        Luis Bernardo added a comment - this patch includes new image handlers for PNG for PDF and PS output. these image handlers can handle the raw IDAT chunk and lead to smaller files. to use these handlers the ImageLoaderRawPNG needs to be enabled in the fop.xconf file.
        Hide
        Luis Bernardo added a comment -

        Attachment apachefop.patch has been added with description: patch

        Show
        Luis Bernardo added a comment - Attachment apachefop.patch has been added with description: patch
        Hide
        Luis Bernardo added a comment -

        the attachment includes a test.fo file, some test images, a fop.xconf, and PDF and PS output of the test file. note that the image loader in the fop.xconf needs to be enabled/disabled.

        Show
        Luis Bernardo added a comment - the attachment includes a test.fo file, some test images, a fop.xconf, and PDF and PS output of the test file. note that the image loader in the fop.xconf needs to be enabled/disabled.
        Hide
        Luis Bernardo added a comment -

        Attachment test.tar.gz has been added with description: test examples

        Show
        Luis Bernardo added a comment - Attachment test.tar.gz has been added with description: test examples
        Hide
        Luis Bernardo added a comment -

        this example uses a larger image to better show the differences in file size of the resulting PDF.

        Show
        Luis Bernardo added a comment - this example uses a larger image to better show the differences in file size of the resulting PDF.
        Hide
        Luis Bernardo added a comment -

        Attachment example.tar.gz has been added with description: another example

        Show
        Luis Bernardo added a comment - Attachment example.tar.gz has been added with description: another example
        Hide
        Glenn Adams added a comment -

        patch applied at http://svn.apache.org/viewvc?rev=1350455&view=rev

        thanks luis and matthias!

        luis, please provide an additional patch (against this rev) that fixes the 13 findbugs warnings you introduced (I fixed the 2 checkstyle warnings introduced); also update the relevant documentation to describe the configuration/use of these features

        in the mean time, I will transition this bug to NEEDINFO

        Show
        Glenn Adams added a comment - patch applied at http://svn.apache.org/viewvc?rev=1350455&view=rev thanks luis and matthias! luis, please provide an additional patch (against this rev) that fixes the 13 findbugs warnings you introduced (I fixed the 2 checkstyle warnings introduced); also update the relevant documentation to describe the configuration/use of these features in the mean time, I will transition this bug to NEEDINFO
        Hide
        Luis Bernardo added a comment -

        fixes/ignores findbugs

        Show
        Luis Bernardo added a comment - fixes/ignores findbugs
        Hide
        Luis Bernardo added a comment -

        Attachment bug40676-findbugs.patch has been added with description: patch for findbugs

        Show
        Luis Bernardo added a comment - Attachment bug40676-findbugs.patch has been added with description: patch for findbugs
        Show
        Luis Bernardo added a comment - Wiki updated: http://wiki.apache.org/xmlgraphics-fop/HowTo/ImageLoaderRawPNG
        Hide
        Glenn Adams added a comment -

        findbugs fix patch applied at http://svn.apache.org/viewvc?rev=1350790&view=rev

        thanks luis! please review and close if satisfied

        Show
        Glenn Adams added a comment - findbugs fix patch applied at http://svn.apache.org/viewvc?rev=1350790&view=rev thanks luis! please review and close if satisfied
        Hide
        Glenn Adams added a comment -

        (In reply to comment #13)
        > Wiki updated: http://wiki.apache.org/xmlgraphics-fop/HowTo/ImageLoaderRawPNG

        you may also want to update [1] and [2]

        [1] http://xmlgraphics.apache.org/fop/trunk/graphics.html#png
        [2] http://xmlgraphics.apache.org/fop/trunk/configuration.html#image-loading

        for example, adding info to [1] about the new codecs, and adding info to [2] about negative penalties

        Show
        Glenn Adams added a comment - (In reply to comment #13) > Wiki updated: http://wiki.apache.org/xmlgraphics-fop/HowTo/ImageLoaderRawPNG you may also want to update [1] and [2] [1] http://xmlgraphics.apache.org/fop/trunk/graphics.html#png [2] http://xmlgraphics.apache.org/fop/trunk/configuration.html#image-loading for example, adding info to [1] about the new codecs, and adding info to [2] about negative penalties
        Hide
        Luis Bernardo added a comment -

        I was not aware the web pages content was part of the source too... The patch updates the documentation.

        Show
        Luis Bernardo added a comment - I was not aware the web pages content was part of the source too... The patch updates the documentation.
        Hide
        Luis Bernardo added a comment -

        Attachment fop-png-documentation.patch has been added with description: patch for documentation

        Show
        Luis Bernardo added a comment - Attachment fop-png-documentation.patch has been added with description: patch for documentation
        Hide
        Glenn Adams added a comment -

        (In reply to comment #16)
        > Created attachment 28950 [details]
        > patch for documentation
        >
        > I was not aware the web pages content was part of the source too... The
        > patch updates the documentation.

        Applied final docs patch at http://svn.apache.org/viewvc?rev=1351540&view=rev.

        Thanks luis! Please close this bug when you finish review.

        Show
        Glenn Adams added a comment - (In reply to comment #16) > Created attachment 28950 [details] > patch for documentation > > I was not aware the web pages content was part of the source too... The > patch updates the documentation. Applied final docs patch at http://svn.apache.org/viewvc?rev=1351540&view=rev . Thanks luis! Please close this bug when you finish review.
        Hide
        Luis Bernardo added a comment -

        Adobe Reader fails to display the submitted examples when the raw png image loader is used, even though they seem valid according to the spec, and are correctly displayed by evince, ghostscript and Preview (Mac OS X PDF viewer). The Adobe Reader issue only happens if the filter flate is turned on and seems to be due to the fact that Adobe Reader may not like more than one filter applied to an image. Multiple filters applied to a stream are valid, but are also used by PDF malware exploits (like zip bombs). I suspect this is the reason why Adobe Reader does not like them.

        The patch fixes this problem by only using one image filter even when the default flate filter is turned on.

        Show
        Luis Bernardo added a comment - Adobe Reader fails to display the submitted examples when the raw png image loader is used, even though they seem valid according to the spec, and are correctly displayed by evince, ghostscript and Preview (Mac OS X PDF viewer). The Adobe Reader issue only happens if the filter flate is turned on and seems to be due to the fact that Adobe Reader may not like more than one filter applied to an image. Multiple filters applied to a stream are valid, but are also used by PDF malware exploits (like zip bombs). I suspect this is the reason why Adobe Reader does not like them. The patch fixes this problem by only using one image filter even when the default flate filter is turned on.
        Hide
        Luis Bernardo added a comment -

        Attachment fop-disallowmultiplefilters.patch has been added with description: patch to disallow multiple image filters

        Show
        Luis Bernardo added a comment - Attachment fop-disallowmultiplefilters.patch has been added with description: patch to disallow multiple image filters
        Hide
        Luis Bernardo added a comment -

        Attachment test.tar.gz has been added with description: update to previous test examples

        Show
        Luis Bernardo added a comment - Attachment test.tar.gz has been added with description: update to previous test examples
        Hide
        Luis Bernardo added a comment -

        Attachment example.tar.gz has been added with description: update to previous example

        Show
        Luis Bernardo added a comment - Attachment example.tar.gz has been added with description: update to previous example
        Hide
        Glenn Adams added a comment -

        (In reply to comment #18)
        > Created attachment 28988 [details]
        > patch to disallow multiple image filters
        >
        > Adobe Reader fails to display the submitted examples when the raw png image
        > loader is used, even though they seem valid according to the spec, and are
        > correctly displayed by evince, ghostscript and Preview (Mac OS X PDF
        > viewer). The Adobe Reader issue only happens if the filter flate is turned
        > on and seems to be due to the fact that Adobe Reader may not like more than
        > one filter applied to an image. Multiple filters applied to a stream are
        > valid, but are also used by PDF malware exploits (like zip bombs). I suspect
        > this is the reason why Adobe Reader does not like them.
        >
        > The patch fixes this problem by only using one image filter even when the
        > default flate filter is turned on.

        patch applied at http://svn.apache.org/viewvc?rev=1353303&view=rev

        thanks luis!

        Show
        Glenn Adams added a comment - (In reply to comment #18) > Created attachment 28988 [details] > patch to disallow multiple image filters > > Adobe Reader fails to display the submitted examples when the raw png image > loader is used, even though they seem valid according to the spec, and are > correctly displayed by evince, ghostscript and Preview (Mac OS X PDF > viewer). The Adobe Reader issue only happens if the filter flate is turned > on and seems to be due to the fact that Adobe Reader may not like more than > one filter applied to an image. Multiple filters applied to a stream are > valid, but are also used by PDF malware exploits (like zip bombs). I suspect > this is the reason why Adobe Reader does not like them. > > The patch fixes this problem by only using one image filter even when the > default flate filter is turned on. patch applied at http://svn.apache.org/viewvc?rev=1353303&view=rev thanks luis!
        Hide
        Glenn Adams added a comment -

        (In reply to comment #19)
        > Created attachment 28989 [details]
        > update to previous test examples

        not sure what to do with these... do you wish to commit to trunk somewhere? if so, you need to organize into standard test cases, etc

        Show
        Glenn Adams added a comment - (In reply to comment #19) > Created attachment 28989 [details] > update to previous test examples not sure what to do with these... do you wish to commit to trunk somewhere? if so, you need to organize into standard test cases, etc
        Hide
        Luis Bernardo added a comment -

        no, the idea was never to make these part of the source. they are just provided to show what type of png images are currently supported by the new png image loader, and that the resulting output files are smaller.

        Show
        Luis Bernardo added a comment - no, the idea was never to make these part of the source. they are just provided to show what type of png images are currently supported by the new png image loader, and that the resulting output files are smaller.
        Hide
        Glenn Adams added a comment -

        (In reply to comment #23)
        > no, the idea was never to make these part of the source. they are just
        > provided to show what type of png images are currently supported by the new
        > png image loader, and that the resulting output files are smaller.

        ok, if you don't think any additional test cases are useful for this bug, then please close this when you're ready

        Show
        Glenn Adams added a comment - (In reply to comment #23) > no, the idea was never to make these part of the source. they are just > provided to show what type of png images are currently supported by the new > png image loader, and that the resulting output files are smaller. ok, if you don't think any additional test cases are useful for this bug, then please close this when you're ready
        Hide
        Luis Bernardo added a comment -

        this new patch improves the previous one and adds support to color spaces to the PDFImageHandlerRawPNG.

        Show
        Luis Bernardo added a comment - this new patch improves the previous one and adds support to color spaces to the PDFImageHandlerRawPNG.
        Hide
        Luis Bernardo added a comment -

        Attachment fop-bug40676.patch has been added with description: support for sRGB and iCCP chunks

        Show
        Luis Bernardo added a comment - Attachment fop-bug40676.patch has been added with description: support for sRGB and iCCP chunks
        Hide
        Luis Bernardo added a comment -

        Attachment test-iccp.pdf has been added with description: example with ICC profiles

        Show
        Luis Bernardo added a comment - Attachment test-iccp.pdf has been added with description: example with ICC profiles
        Hide
        Luis Bernardo added a comment -

        Attachment example_images_iccp.tar.gz has been added with description: example images with ICC profiles

        Show
        Luis Bernardo added a comment - Attachment example_images_iccp.tar.gz has been added with description: example images with ICC profiles
        Hide
        Luis Bernardo added a comment -

        Attachment test-srgb-rendering_intent.pdf has been added with description: example with sRGB profile (sRGB chunk)

        Show
        Luis Bernardo added a comment - Attachment test-srgb-rendering_intent.pdf has been added with description: example with sRGB profile (sRGB chunk)
        Hide
        Luis Bernardo added a comment -

        Attachment example_images_srgb.tar.gz has been added with description: example images with sRGB chunk

        Show
        Luis Bernardo added a comment - Attachment example_images_srgb.tar.gz has been added with description: example images with sRGB chunk
        Hide
        Ognjen Blagojevic added a comment -

        Although file size increase is now solved, I am reopening this issue as a reminder to apply patches starting with attachment 29132.

        Those new patches perform better, and they also solve issue 51149.

        Show
        Ognjen Blagojevic added a comment - Although file size increase is now solved, I am reopening this issue as a reminder to apply patches starting with attachment 29132. Those new patches perform better, and they also solve issue 51149.
        Hide
        Luis Bernardo added a comment -
        Show
        Luis Bernardo added a comment - applied patch 29132 with changes: http://svn.apache.org/viewvc?view=revision&revision=1400536

          People

          • Assignee:
            fop-dev
            Reporter:
            Paul A. Bristow
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development