Uploaded image for project: 'Groovy'
  1. Groovy
  2. GROOVY-10232

Massive increase in memory usage due to CacheableCallSite

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.9
    • 3.0.10, 4.0.0
    • groovy-runtime
    • None

    Description

      When upgrading our tests from Spock 1.3 with Groovy 2.5 to Spock 2.0 with Groovy 3.0.9, we are seeing issue with memory usage caused by CacheableCallSite. This memory seems to be retained in classes and is therefore never freed. A single Spock test class can take as much as 150mb memory. The total amount of memory sums up to several gigabytes of additional memory, causing our tests to take about 3 times as much memory as with Spock 1.3 and Groovy 2.5.

      Attachments

        1. Screenshot_20210913_161502.png
          209 kB
          Emond Papegaaij

        Issue Links

          Activity

            daniel_sun Daniel Sun added a comment - https://github.com/apache/groovy/pull/1647
            daniel_sun Daniel Sun added a comment -

            If you could run your tests with the following snapshot of Groovy 4 and give us some feedback, that would be great.
            https://github.com/apache/groovy/actions/runs/1375230234

            daniel_sun Daniel Sun added a comment - If you could run your tests with the following snapshot of Groovy 4 and give us some feedback, that would be great. https://github.com/apache/groovy/actions/runs/1375230234

            Our tests are built with Spock, which is not compatible with groovy 4. Any chance this will be backported to 3? I'd be more than happy to give it a try if you can provide me with maven artifacts for a patched Groovy 3 SNAPSHOT. The link to the SNAPSHOT build on github does not seem to contain any maven artifacts.

            papegaaij Emond Papegaaij added a comment - Our tests are built with Spock, which is not compatible with groovy 4. Any chance this will be backported to 3? I'd be more than happy to give it a try if you can provide me with maven artifacts for a patched Groovy 3 SNAPSHOT. The link to the SNAPSHOT build on github does not seem to contain any maven artifacts.
            paulk Paul King added a comment - - edited

            papegaaij Just FYI, there is a switch for Spock for working with Groovy 4 - it is used by the Groovy build for instance.
            https://github.com/apache/groovy/blob/master/buildSrc/src/main/groovy/org.apache.groovy-base.gradle#L220

            paulk Paul King added a comment - - edited papegaaij Just FYI, there is a switch for Spock for working with Groovy 4 - it is used by the Groovy build for instance. https://github.com/apache/groovy/blob/master/buildSrc/src/main/groovy/org.apache.groovy-base.gradle#L220

            paulk It's not just Spock, but our entire infrastructure to run tests. Our tests are based on Arquillian, so for Groovy 4 to work, I will need to update our integration for Spock and Arquillian (https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5) as well. Also, because the maven coordinates have changed, this will need to be changed in our packaging as well, because Arquillian will need to know which classes to include with the tests to run. As you might understand, this will require a significant effort and the results will not be entirely reliable, because it's all based on experimental releases and things that are not built to work together.

            The patch applies cleanly on the GROOVY_3.0.X branch, so I did a run with that version. It looks like the problem is fixed. I've reduced the memory settings to the values we used for Groovy 2.5 and no longer get an OutOfMemoryError.

            papegaaij Emond Papegaaij added a comment - paulk It's not just Spock, but our entire infrastructure to run tests. Our tests are based on Arquillian, so for Groovy 4 to work, I will need to update our integration for Spock and Arquillian ( https://github.com/topicusonderwijs/arquillian-testrunner-spock/tree/spock-2.0-junit5 ) as well. Also, because the maven coordinates have changed, this will need to be changed in our packaging as well, because Arquillian will need to know which classes to include with the tests to run. As you might understand, this will require a significant effort and the results will not be entirely reliable, because it's all based on experimental releases and things that are not built to work together. The patch applies cleanly on the GROOVY_3.0.X branch, so I did a run with that version. It looks like the problem is fixed. I've reduced the memory settings to the values we used for Groovy 2.5 and no longer get an OutOfMemoryError.
            ian.springer Ian Springer added a comment -

            My team also encountered this issue with our Groovy test suites after upgrading from Groovy 2.x to Groovy 3.0.9. We are running our tests on GraalVM 21.2.0 (JDK 11.0.12) with TestNG 6.14.3. We increased the max heap size for our integration test suite from 4 GB to 18 GB and still ran out of heap. At the point when the JVM ran out of heap there were 108K instances of CacheableCallSite consuming 18.5 GB (96%) of the heap.

             

            We are going to try the patch from PR 1647, but would prefer not having to fork Groovy to work around this issue. We noticed PR 1647 has been closed. Does the Groovy team still plan on addressing this issue?

             

            Thanks!

            ian.springer Ian Springer added a comment - My team also encountered this issue with our Groovy test suites after upgrading from Groovy 2.x to Groovy 3.0.9. We are running our tests on GraalVM 21.2.0 (JDK 11.0.12) with TestNG 6.14.3. We increased the max heap size for our integration test suite from 4 GB to 18 GB and still ran out of heap. At the point when the JVM ran out of heap there were 108K instances of CacheableCallSite consuming 18.5 GB (96%) of the heap.   We are going to try the patch from PR 1647, but would prefer not having to fork Groovy to work around this issue. We noticed PR 1647 has been closed. Does the Groovy team still plan on addressing this issue?   Thanks!
            daniel_sun Daniel Sun added a comment - Fixed by https://github.com/apache/groovy/commit/66bb24acabe6a7a700c3d09c8fa0bc6b236d535b
            daniel_sun Daniel Sun added a comment -

            ian.springer papegaaij 

            Please give it a try, if the issue is solved, I'll close it:

            https://github.com/apache/groovy/actions/runs/1671424767

            daniel_sun Daniel Sun added a comment - ian.springer papegaaij   Please give it a try, if the issue is solved, I'll close it: https://github.com/apache/groovy/actions/runs/1671424767
            ian.springer Ian Springer added a comment -

            daniel_sun, it looks like that did the trick. Thanks!

            ian.springer Ian Springer added a comment - daniel_sun , it looks like that did the trick. Thanks!

            I can also confirm that the memory usage has dropped back to what it was in groovy 2. Do you have any idea when we can expect a 3.0.10 release with this fix?

            papegaaij Emond Papegaaij added a comment - I can also confirm that the memory usage has dropped back to what it was in groovy 2. Do you have any idea when we can expect a 3.0.10 release with this fix?
            daniel_sun Daniel Sun added a comment -

            Thanks for your feedback.

            As for 3.0.10 release plan, paulk can tell us

            daniel_sun Daniel Sun added a comment - Thanks for your feedback. As for 3.0.10 release plan, paulk can tell us
            paulk Paul King added a comment -

            I will do the release for 3.0.10 at the same time or shortly after 4 GA.

            paulk Paul King added a comment - I will do the release for 3.0.10 at the same time or shortly after 4 GA.
            ian.springer Ian Springer added a comment -

            paulk, when do you think 4 will go GA? Thanks!

            ian.springer Ian Springer added a comment - paulk , when do you think 4 will go GA? Thanks!
            chadwilson Chad Wilson added a comment -

            Thanks for the work here! I've been scratching my head about which change has caused instability on builds for https://github.com/gocd/gocd/ for some time, but suspect this one.

            Does anyone know if this is likely to have affected all 3.0.x versions, or only a particular point/patch release - i.e could a downgrade to an earlier 3.0.x version help?

            chadwilson Chad Wilson added a comment - Thanks for the work here! I've been scratching my head about which change has caused instability on builds for https://github.com/gocd/gocd/ for some time, but suspect this one. Does anyone know if this is likely to have affected all 3.0.x versions, or only a particular point/patch release - i.e could a downgrade to an earlier 3.0.x version help?
            chadwilson Chad Wilson added a comment -

            I see 4.0.0 is GA as of 11 days ago (congratulations!), and there is quite a bit work on 3.0.10 going on based on https://github.com/apache/groovy/commits/GROOVY_3_0_X - hopefully we are close to a release here

            chadwilson Chad Wilson added a comment - I see 4.0.0 is GA as of 11 days ago (congratulations!), and there is quite a bit work on 3.0.10 going on based on https://github.com/apache/groovy/commits/GROOVY_3_0_X - hopefully we are close to a release here

            People

              daniel_sun Daniel Sun
              papegaaij Emond Papegaaij
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h