Fop
  1. Fop
  2. FOP-1605

OutOfMemoryError after generating several thousand pdfs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Resolution: Fixed
    • Affects Version/s: trunk
    • Fix Version/s: None
    • Component/s: fo/unqualified
    • Labels:
      None
    • Environment:
      Operating System: All
      Platform: All
    • External issue ID:
      46319

      Description

      After upgrading from fop 0.20.5 to fop 0.95 our web application crashes after generating about 15000 small size pdfs (max. 1-12 pages) with the following exception: (number of bytes varies)

      Exception java.lang.OutOfMemoryError: requested 65536000 bytes for GrET* in /BUILD_AREA/jdk1.5.0_10/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?

      Before crashing used heap and process memory are growing slowly but continously. (USER_MEM_ARGS="-Xmx3072m -XX:PermSize=128m")

      We already tried using newest xalan/xerces libraries (from 2.7.1 xalan distribution) and also applied bug fix from http://svn.apache.org/viewvc?view=rev&revision=698322 to our fop o.95 library, but this made no difference. And also using a fop trunk version built end of october did not help.

      Profiling results regarding potential memory leaks are not yet available.

      1. profileSnapshot.JPG
        236 kB
        Adrian Meyer
      2. heapWalker3.JPG
        77 kB
        Adrian Meyer
      3. bug46319.diff
        4 kB
        Andreas L. Delmelle
      4. memleakPatches.diff
        84 kB
        Adrian Meyer

        Activity

        Hide
        Jeremias Maerki added a comment -

        I'm not sure this is a FOP problem. The error message says:

        > Exception java.lang.OutOfMemoryError: requested 65536000 bytes for GrET* in
        > /BUILD_AREA/jdk1.5.0_10/hotspot/src/share/vm/utilities/growableArray.cpp. Out
        > of swap space?

        If the VM says it's out of swap space, it doesn't get enough memory from the operating system. Are you on a 32bit or 64 bit Linux? At any rate, I'd limit the maximum memory to 2GB (or less) for the moment and check what happens. Why did you give the JVM so much memory to begin with? Documents with up to 12 pages are not likely to need that amount of memory except maybe if you're trying to process huge images.

        You say FOP is working in a web application. In that case it can make sense to limit the number of concurrent processing runs to a manageable level (also as a DDoS measure).

        That said, I've made extensive memory usage tests for FOP 0.95 and detected no memory leaks. I'm inclined to close this issue as invalid unless you have additional information indicating a problem in FOP.

        Show
        Jeremias Maerki added a comment - I'm not sure this is a FOP problem. The error message says: > Exception java.lang.OutOfMemoryError: requested 65536000 bytes for GrET* in > /BUILD_AREA/jdk1.5.0_10/hotspot/src/share/vm/utilities/growableArray.cpp. Out > of swap space? If the VM says it's out of swap space, it doesn't get enough memory from the operating system. Are you on a 32bit or 64 bit Linux? At any rate, I'd limit the maximum memory to 2GB (or less) for the moment and check what happens. Why did you give the JVM so much memory to begin with? Documents with up to 12 pages are not likely to need that amount of memory except maybe if you're trying to process huge images. You say FOP is working in a web application. In that case it can make sense to limit the number of concurrent processing runs to a manageable level (also as a DDoS measure). That said, I've made extensive memory usage tests for FOP 0.95 and detected no memory leaks. I'm inclined to close this issue as invalid unless you have additional information indicating a problem in FOP.
        Hide
        Chris Bowditch added a comment -

        Jeremias,

        what about the memory leak in the property cache which you fixed in Trunk since 0.95, as detailed below:

        http://xmlgraphics.apache.org/fop/changes.html

        Show
        Chris Bowditch added a comment - Jeremias, what about the memory leak in the property cache which you fixed in Trunk since 0.95, as detailed below: http://xmlgraphics.apache.org/fop/changes.html
        Hide
        Adrian Meyer added a comment -

        Attachment profileSnapshot.JPG has been added with description: growing allocation area

        Show
        Adrian Meyer added a comment - Attachment profileSnapshot.JPG has been added with description: growing allocation area
        Hide
        Adrian Meyer added a comment -

        The webapplication runs on 64 bit linux. The 3GB memory are needed to run about 300 parallel sessions with about 1000 request per minute, where as 10% are pdf requests and the rest http.

        We already limit the maximum number of parallel fop requests by using a pool of 3 pfdTransformer objects.

        We blame fop 0.95 to be problem cause because the application runs without problems when turning of the pdf requests or when using old fop 0.20.5 for pdf generation.

        I just started some investigations with profiler - it looks like a few kb (~2kB) of memory get lost per pdf. I hope the attached screenshot helps to find the potential problem cause.

        Show
        Adrian Meyer added a comment - The webapplication runs on 64 bit linux. The 3GB memory are needed to run about 300 parallel sessions with about 1000 request per minute, where as 10% are pdf requests and the rest http. We already limit the maximum number of parallel fop requests by using a pool of 3 pfdTransformer objects. We blame fop 0.95 to be problem cause because the application runs without problems when turning of the pdf requests or when using old fop 0.20.5 for pdf generation. I just started some investigations with profiler - it looks like a few kb (~2kB) of memory get lost per pdf. I hope the attached screenshot helps to find the potential problem cause.
        Hide
        Jeremias Maerki added a comment -

        (In reply to comment #2)
        > Jeremias,
        >
        > what about the memory leak in the property cache which you fixed in Trunk since
        > 0.95, as detailed below:
        >
        > http://xmlgraphics.apache.org/fop/changes.html
        >

        Right, forgot about that one. I'm just not sure after all the changes in PropertyCache when memory leaks were introduced and when they were fixed again (looking at the log for that file shows we had quite a few attempts to get that right).

        Show
        Jeremias Maerki added a comment - (In reply to comment #2) > Jeremias, > > what about the memory leak in the property cache which you fixed in Trunk since > 0.95, as detailed below: > > http://xmlgraphics.apache.org/fop/changes.html > Right, forgot about that one. I'm just not sure after all the changes in PropertyCache when memory leaks were introduced and when they were fixed again (looking at the log for that file shows we had quite a few attempts to get that right).
        Hide
        Jeremias Maerki added a comment -

        (In reply to comment #4)
        > The webapplication runs on 64 bit linux. The 3GB memory are needed to run about
        > 300 parallel sessions with about 1000 request per minute, where as 10% are pdf
        > requests and the rest http.
        >
        > We already limit the maximum number of parallel fop requests by using a pool of
        > 3 pfdTransformer objects.
        >
        > We blame fop 0.95 to be problem cause because the application runs without
        > problems when turning of the pdf requests or when using old fop 0.20.5 for pdf
        > generation.
        >
        > I just started some investigations with profiler - it looks like a few kb
        > (~2kB) of memory get lost per pdf. I hope the attached screenshot helps to find
        > the potential problem cause.
        >

        I'm afraid, I can't read anything out of this screenshot. When we fixed the property cache problems, http://www.alphaworks.ibm.com/tech/heapanalyzer was very handy to analyze head dumps and find the leaks.

        Anyway, there's still that strange OutOfMemoryError which I've never seen myself before. Maybe you're seeing two different problems here. I would still check what happens if you lower the max heap setting (if the OutOfMemoryError changes, it's an indicator). And if the OutOfMemoryError persists it can make sense to see if the problem goes away if you use FOP Trunk instead. You can also try to backport the property cache fix to the 0.95 branch but that could be a bit of work.

        Show
        Jeremias Maerki added a comment - (In reply to comment #4) > The webapplication runs on 64 bit linux. The 3GB memory are needed to run about > 300 parallel sessions with about 1000 request per minute, where as 10% are pdf > requests and the rest http. > > We already limit the maximum number of parallel fop requests by using a pool of > 3 pfdTransformer objects. > > We blame fop 0.95 to be problem cause because the application runs without > problems when turning of the pdf requests or when using old fop 0.20.5 for pdf > generation. > > I just started some investigations with profiler - it looks like a few kb > (~2kB) of memory get lost per pdf. I hope the attached screenshot helps to find > the potential problem cause. > I'm afraid, I can't read anything out of this screenshot. When we fixed the property cache problems, http://www.alphaworks.ibm.com/tech/heapanalyzer was very handy to analyze head dumps and find the leaks. Anyway, there's still that strange OutOfMemoryError which I've never seen myself before. Maybe you're seeing two different problems here. I would still check what happens if you lower the max heap setting (if the OutOfMemoryError changes, it's an indicator). And if the OutOfMemoryError persists it can make sense to see if the problem goes away if you use FOP Trunk instead. You can also try to backport the property cache fix to the 0.95 branch but that could be a bit of work.
        Hide
        Adrian Meyer added a comment -

        Reducing heap size did not help - the application died just the same way but much earlier.

        Test with fop.jar from trunk is currently running - does not look very promising according to heap figures.

        I could not find a bug number with the property cache leak fix - is there another way than svn history browsing to find out what was changed for this fix?

        Profiling shows me a growing number of allocations of the following class:
        org.apache.fop.fo.flow.Marker$MarkerAttribute
        Although the instances seem to be referenced from java.util.WeakHashMap only they don't get garbage collected. What are these Marker classes used for and when should they be released again? Is there a way to force their release?

        Show
        Adrian Meyer added a comment - Reducing heap size did not help - the application died just the same way but much earlier. Test with fop.jar from trunk is currently running - does not look very promising according to heap figures. I could not find a bug number with the property cache leak fix - is there another way than svn history browsing to find out what was changed for this fix? Profiling shows me a growing number of allocations of the following class: org.apache.fop.fo.flow.Marker$MarkerAttribute Although the instances seem to be referenced from java.util.WeakHashMap only they don't get garbage collected. What are these Marker classes used for and when should they be released again? Is there a way to force their release?
        Hide
        Adrian Meyer added a comment -

        Allocation call tree is according to attachment 22972

        Show
        Adrian Meyer added a comment - Allocation call tree is according to attachment 22972
        Hide
        Adrian Meyer added a comment -

        Attachment heapWalker3.JPG has been added with description: heap walker incoming references to marker attributes

        Show
        Adrian Meyer added a comment - Attachment heapWalker3.JPG has been added with description: heap walker incoming references to marker attributes
        Hide
        Andreas L. Delmelle added a comment -

        (In reply to comment #7)
        >
        > Profiling shows me a growing number of allocations of the following class:
        > org.apache.fop.fo.flow.Marker$MarkerAttribute
        > Although the instances seem to be referenced from java.util.WeakHashMap only
        > they don't get garbage collected.

        Ouch! Ugly one. Seems to be a classic mistake, again... Since the instances are used both as key and value, they are never released and the map keeps growing.

        > What are these Marker classes used for and when should they be released again?

        For attributes of fo:markers. They should be released as soon as they are no longer referenced.

        > Is there a way to force their release?

        Quick fix would be to modify the code, at line 381 in Marker, and change it to:

        attributeCache.put(new java.lang.ref.WeakReference(newInstance), newInstance);

        Show
        Andreas L. Delmelle added a comment - (In reply to comment #7) > > Profiling shows me a growing number of allocations of the following class: > org.apache.fop.fo.flow.Marker$MarkerAttribute > Although the instances seem to be referenced from java.util.WeakHashMap only > they don't get garbage collected. Ouch! Ugly one. Seems to be a classic mistake, again... Since the instances are used both as key and value, they are never released and the map keeps growing. > What are these Marker classes used for and when should they be released again? For attributes of fo:markers. They should be released as soon as they are no longer referenced. > Is there a way to force their release? Quick fix would be to modify the code, at line 381 in Marker, and change it to: attributeCache.put(new java.lang.ref.WeakReference(newInstance), newInstance);
        Hide
        Andreas L. Delmelle added a comment -

        (In reply to comment #9)
        > (In reply to comment #7)
        > > Is there a way to force their release?
        >
        > Quick fix would be to modify the code, at line 381 in Marker, and change it to:
        >
        > attributeCache.put(new java.lang.ref.WeakReference(newInstance), newInstance);
        >

        Come to think of it, since it was mentioned as a possible cause, the better solution may precisely be to use the PropertyCache here. If MarkerAttribute is made public, then PropertyCache can expose a specialized fetch() for it. In the end, internally, PropertyCache deals with Object, so can be used to serve any type. Given that MarkerAttribute can also be classified as a property-related class, it seems appropriate enough...

        Show
        Andreas L. Delmelle added a comment - (In reply to comment #9) > (In reply to comment #7) > > Is there a way to force their release? > > Quick fix would be to modify the code, at line 381 in Marker, and change it to: > > attributeCache.put(new java.lang.ref.WeakReference(newInstance), newInstance); > Come to think of it, since it was mentioned as a possible cause, the better solution may precisely be to use the PropertyCache here. If MarkerAttribute is made public, then PropertyCache can expose a specialized fetch() for it. In the end, internally, PropertyCache deals with Object, so can be used to serve any type. Given that MarkerAttribute can also be classified as a property-related class, it seems appropriate enough...
        Hide
        Andreas L. Delmelle added a comment -

        Sorry, just realized the proposed quick-fix is bogus :/

        OTOH, changing the MarkerAttributes to use the PropertyCache is relatively simple too... I'll try to attach a patch ASAP.

        Show
        Andreas L. Delmelle added a comment - Sorry, just realized the proposed quick-fix is bogus :/ OTOH, changing the MarkerAttributes to use the PropertyCache is relatively simple too... I'll try to attach a patch ASAP.
        Hide
        Adrian Meyer added a comment -

        (In reply to comment #11)
        > Sorry, just realized the proposed quick-fix is bogus :/
        > OTOH, changing the MarkerAttributes to use the PropertyCache is relatively
        > simple too... I'll try to attach a patch ASAP.

        Due to having a project deadline of end of this week to revert back to fop 0.20.5, if problem is not solved til then, I'd very much appreciate to get a patch very fast. An estimate how many days ASAP may take would be very helpful.

        Show
        Adrian Meyer added a comment - (In reply to comment #11) > Sorry, just realized the proposed quick-fix is bogus :/ > OTOH, changing the MarkerAttributes to use the PropertyCache is relatively > simple too... I'll try to attach a patch ASAP. Due to having a project deadline of end of this week to revert back to fop 0.20.5, if problem is not solved til then, I'd very much appreciate to get a patch very fast. An estimate how many days ASAP may take would be very helpful.
        Hide
        Andreas L. Delmelle added a comment -

        I already did most of it yesterday night, but it was a bit too late to be able to trust myself
        I'll run some tests, and if all goes well, I'll post it later tonight.

        Show
        Andreas L. Delmelle added a comment - I already did most of it yesterday night, but it was a bit too late to be able to trust myself I'll run some tests, and if all goes well, I'll post it later tonight.
        Hide
        Andreas L. Delmelle added a comment -

        (In reply to comment #13)
        > I already did most of it yesterday night, but it was a bit too late to be able
        > to trust myself
        > I'll run some tests, and if all goes well, I'll post it later tonight.
        >

        sigh seems like it would take slightly longer to confirm that nothing breaks, since I've contaminated my sandbox (working on inline-space-resolution...) Due to this unfinished work, the test-suite fails on my end.

        Normally, the patch following shortly should resolve your issue. I'd appreciate if you could confirm that it does.

        Show
        Andreas L. Delmelle added a comment - (In reply to comment #13) > I already did most of it yesterday night, but it was a bit too late to be able > to trust myself > I'll run some tests, and if all goes well, I'll post it later tonight. > sigh seems like it would take slightly longer to confirm that nothing breaks, since I've contaminated my sandbox (working on inline-space-resolution...) Due to this unfinished work, the test-suite fails on my end. Normally, the patch following shortly should resolve your issue. I'd appreciate if you could confirm that it does.
        Hide
        Andreas L. Delmelle added a comment -

        Attachment bug46319.diff has been added with description: Patch that should fix the leaking of Marker$MarkerAttribute (not confirmed)

        Show
        Andreas L. Delmelle added a comment - Attachment bug46319.diff has been added with description: Patch that should fix the leaking of Marker$MarkerAttribute (not confirmed)
        Hide
        Adrian Meyer added a comment -

        Thanks a lot - the patch seems to work. At least I can't spot any leak anymore in the marker area.

        I don't like the idea of going live with a trunk fop version. Would there be a chance to apply this patch to 0.95 source and would this also require to take over the fix for property cache memory leak from jeremias according to comment #5?

        Show
        Adrian Meyer added a comment - Thanks a lot - the patch seems to work. At least I can't spot any leak anymore in the marker area. I don't like the idea of going live with a trunk fop version. Would there be a chance to apply this patch to 0.95 source and would this also require to take over the fix for property cache memory leak from jeremias according to comment #5?
        Hide
        Andreas L. Delmelle added a comment -

        (In reply to comment #16)
        > Thanks a lot - the patch seems to work. At least I can't spot any leak anymore
        > in the marker area.
        >
        > I don't like the idea of going live with a trunk fop version. Would there be a
        > chance to apply this patch to 0.95 source and would this also require to take
        > over the fix for property cache memory leak from jeremias according to comment
        > #5?

        I think so, yes. You could apply those patches to 0.95 with the same effect (only the line-numbers will be slightly off, due to other changes in FOP Trunk)
        To be entirely complete, you'll probably have to include some other commits as well. After the release of 0.95, caching has also been applied to some other property-related classes (org.apache.fop.fo.properties.CommonBorderPaddingBackground being the most notable).

        I'll look up the exact commit for the fix of the leak in PropertyCache, and will post it here when found.

        Show
        Andreas L. Delmelle added a comment - (In reply to comment #16) > Thanks a lot - the patch seems to work. At least I can't spot any leak anymore > in the marker area. > > I don't like the idea of going live with a trunk fop version. Would there be a > chance to apply this patch to 0.95 source and would this also require to take > over the fix for property cache memory leak from jeremias according to comment > #5? I think so, yes. You could apply those patches to 0.95 with the same effect (only the line-numbers will be slightly off, due to other changes in FOP Trunk) To be entirely complete, you'll probably have to include some other commits as well. After the release of 0.95, caching has also been applied to some other property-related classes (org.apache.fop.fo.properties.CommonBorderPaddingBackground being the most notable). I'll look up the exact commit for the fix of the leak in PropertyCache, and will post it here when found.
        Hide
        Andreas L. Delmelle added a comment -

        (In reply to comment #17)
        >
        > I'll look up the exact commit for the fix of the leak in PropertyCache, and
        > will post it here when found.
        >

        That was easy. The last commit made to that class was three months ago, and contained the fix in question, see:

        http://svn.apache.org/viewvc/xmlgraphics/fop/trunk/src/java/org/apache/fop/fo/properties/PropertyCache.java?view=log

        Show
        Andreas L. Delmelle added a comment - (In reply to comment #17) > > I'll look up the exact commit for the fix of the leak in PropertyCache, and > will post it here when found. > That was easy. The last commit made to that class was three months ago, and contained the fix in question, see: http://svn.apache.org/viewvc/xmlgraphics/fop/trunk/src/java/org/apache/fop/fo/properties/PropertyCache.java?view=log
        Hide
        Adrian Meyer added a comment -

        (In reply to comment #18)
        > (In reply to comment #17)
        > >
        > > I'll look up the exact commit for the fix of the leak in PropertyCache, and
        > > will post it here when found.
        > >
        > That was easy. The last commit made to that class was three months ago, and
        > contained the fix in question, see:
        >
        No - it wasn't that easy due to 0.95 src having been branched already end of last year. So a few more revisions and files had to be taken into account.

        diff with all the changes applied to 0.95 source will be added on Monday (if it proves to be working til then)

        Show
        Adrian Meyer added a comment - (In reply to comment #18) > (In reply to comment #17) > > > > I'll look up the exact commit for the fix of the leak in PropertyCache, and > > will post it here when found. > > > That was easy. The last commit made to that class was three months ago, and > contained the fix in question, see: > No - it wasn't that easy due to 0.95 src having been branched already end of last year. So a few more revisions and files had to be taken into account. diff with all the changes applied to 0.95 source will be added on Monday (if it proves to be working til then)
        Hide
        Adrian Meyer added a comment -

        Includes following bug fixes

        Show
        Adrian Meyer added a comment - Includes following bug fixes patch from current issue PropertyCache memory leak fix Memory leak in XMLWhiteSpaceHandler ( http://svn.apache.org/viewvc?view=rev&revision=698322 ) (not memory leak) fix StackOverflow on TableColumn (issue 45862)
        Hide
        Adrian Meyer added a comment -

        Attachment memleakPatches.diff has been added with description: Trunk patches applied to 0.95 source

        Show
        Adrian Meyer added a comment - Attachment memleakPatches.diff has been added with description: Trunk patches applied to 0.95 source
        Hide
        Adrian Meyer added a comment -

        test with patched 0.95 version run ok for more than 60 hours - no more leak detected.

        Best thanks once again for quickly solving this issue.

        Show
        Adrian Meyer added a comment - test with patched 0.95 version run ok for more than 60 hours - no more leak detected. Best thanks once again for quickly solving this issue.
        Hide
        Andreas L. Delmelle added a comment -

        (In reply to comment #21)
        > test with patched 0.95 version run ok for more than 60 hours - no more leak
        > detected.
        >
        > Best thanks once again for quickly solving this issue.

        OK, thanks for the confirmation.
        Remaining patch, to fix the MarkerAttribute leak, applied to FOP Trunk with r724444.

        Thanks also for providing the patch to the 0.95 branch. I'm not going to applying it myself, but it could prove of use to others.

        Show
        Andreas L. Delmelle added a comment - (In reply to comment #21) > test with patched 0.95 version run ok for more than 60 hours - no more leak > detected. > > Best thanks once again for quickly solving this issue. OK, thanks for the confirmation. Remaining patch, to fix the MarkerAttribute leak, applied to FOP Trunk with r724444. Thanks also for providing the patch to the 0.95 branch. I'm not going to applying it myself, but it could prove of use to others.
        Hide
        Glenn Adams added a comment -

        batch transition pre-FOP1.0 resolved+fixed bugs to closed+fixed

        Show
        Glenn Adams added a comment - batch transition pre-FOP1.0 resolved+fixed bugs to closed+fixed

          People

          • Assignee:
            fop-dev
            Reporter:
            Adrian Meyer
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development