Uploaded image for project: 'Maven'
  1. Maven
  2. MNG-6030

ReactorModelCache do not used effectively after maven version 3.0.5 which cause a large memory footprint

    Details

      Description

      after version 3.0.5 we found a very big memory consumption with maven. needed for build memory grows from 1G to 5G

      According to current implementation there were change with commit https://git-wip-us.apache.org/repos/asf?p=maven.git;a=commit;h=e778ea67121f17232a7dced3fb4f0a205ffa64af
      https://issues.apache.org/jira/browse/MNG-5312

      private ModelBuildingRequest getModelBuildingRequest( InternalConfig config )
          {
      @@ -256,7 +256,7 @@ public class DefaultProjectBuilder
               request.setUserProperties( configuration.getUserProperties() );
               request.setBuildStartTime( configuration.getBuildStartTime() );
               request.setModelResolver( resolver );
      -        request.setModelCache( config.modelCache );
      +        request.setModelCache( new ReactorModelCache() );
      
               return request;
           }
      

      and as result new ReactorModelCache created for each ModelBuildingRequest which is created new to parse each pom.xml included in the build as module.

      so if there is a big projects with a lot of modules ModelBuildingRequest is created for each of them and parse result is not reused between modules. and if all this modules has the same parent pom it parsed each time and stored as many times as many ModelBuildingRequest was created. As parent pom often contains dependencies, dependencyManagement and common information for all modules it cause a larger memory footprint then was before. In version 3.0.5 and earlier ReactorModelCache was single and was reused between ModelBuildingRequest.

      in our cause footprint grew from 1g to 5g.

      it will be better to reuse result of parsing pom files between ModelBuildingRequests.

      1. generate.heavy.parent.pom.and.deep.hierarchy.groovy
        6 kB
        Andriy
      2. generate.heavy.parent.pom.groovy
        6 kB
        Andriy
      3. screenshot-maven-3.0.5.png
        790 kB
        Karl Heinz Marbaise
      4. screenshot-maven-3.3.9.png
        826 kB
        Karl Heinz Marbaise
      5. screenshot-maven-3.3.9-details.png
        808 kB
        Karl Heinz Marbaise

        Issue Links

          Activity

          Hide
          khmarbaise Karl Heinz Marbaise added a comment - - edited

          Hi there is thread on the dev list on going about the subject..furthermore i have a test setup which creates a multi module reactor with 5000 modules for testing...
          Apart from that i have made several heap dumps with the test projects which shows me a different cause...but we might combine that and see where the real cause is located..

          I have attached screenshots of the heap dump analyses. They show for maven 3.3.9 there are a large amount of instances of this: DefaultProjectBuilder$InterimResult.

          Show
          khmarbaise Karl Heinz Marbaise added a comment - - edited Hi there is thread on the dev list on going about the subject..furthermore i have a test setup which creates a multi module reactor with 5000 modules for testing... Apart from that i have made several heap dumps with the test projects which shows me a different cause...but we might combine that and see where the real cause is located.. I have attached screenshots of the heap dump analyses. They show for maven 3.3.9 there are a large amount of instances of this: DefaultProjectBuilder$InterimResult .
          Hide
          sandriy Andriy added a comment - - edited

          What i found while my investigation:
          I locally added logging in the cache class to calculate cache miss and found the cache in not used for parent pom.
          In our project I made local change for sharing the cache between modules and it cause smaller footprint.(the same as was in maven 3.0.5)

          according to the DefaultProjectBuilder$InterimResult . it contains ModelBuildingResultinside it

                  ModelBuildingRequest request = getModelBuildingRequest( config );
          ....
                      ModelBuildingResult result = modelBuilder.build( request );
          ...
                      InterimResult interimResult = new InterimResult( pomFile, request, result, listener, isRoot );
          

          and each ModelBuildingRequest has separate instance of the cache inside it

                  request.setModelCache( new ReactorModelCache() );
          

          so now we has as many InterimResult as we has modules and because of the not reused caches also so many parsed parent poms in each cache.
          Also we has a large cache miss value while build.

          reusing ReactorModelCache will cause smaller footprint of the interimResult and smaller cache miss value for the cache too.

          Show
          sandriy Andriy added a comment - - edited What i found while my investigation: I locally added logging in the cache class to calculate cache miss and found the cache in not used for parent pom. In our project I made local change for sharing the cache between modules and it cause smaller footprint.(the same as was in maven 3.0.5) according to the DefaultProjectBuilder$InterimResult . it contains ModelBuildingResultinside it ModelBuildingRequest request = getModelBuildingRequest( config ); .... ModelBuildingResult result = modelBuilder.build( request ); ... InterimResult interimResult = new InterimResult( pomFile, request, result, listener, isRoot ); and each ModelBuildingRequest has separate instance of the cache inside it request.setModelCache( new ReactorModelCache() ); so now we has as many InterimResult as we has modules and because of the not reused caches also so many parsed parent poms in each cache. Also we has a large cache miss value while build. reusing ReactorModelCache will cause smaller footprint of the interimResult and smaller cache miss value for the cache too.
          Hide
          sandriy Andriy added a comment -

          i have reviewed the groovy script you provided to generate maven model and found that the parent pom is too lightweight.

          i made changes to generated maven project adding dependency management section in parent pom
          and in second file i made more deep hierarchy when each level module as a parent have previous created module.

          both groovy scripts i am attaching to the ticket.

          Show
          sandriy Andriy added a comment - i have reviewed the groovy script you provided to generate maven model and found that the parent pom is too lightweight. i made changes to generated maven project adding dependency management section in parent pom and in second file i made more deep hierarchy when each level module as a parent have previous created module. both groovy scripts i am attaching to the ticket.
          Hide
          khmarbaise Karl Heinz Marbaise added a comment -

          You could have made a pull request to the git project to enhance the groovy script...

          Show
          khmarbaise Karl Heinz Marbaise added a comment - You could have made a pull request to the git project to enhance the groovy script...
          Hide
          sandriy Andriy added a comment -

          ok. i made pull request as you sugested

          Show
          sandriy Andriy added a comment - ok. i made pull request as you sugested
          Hide
          khmarbaise Karl Heinz Marbaise added a comment -

          Andriy Yes you are right about the lightweightness. But having 5000 modules shows the problem using a simple mvn clean and shows the success of your investigations. I have created a patch based on your suggestions. Many thanks for your help and investigation.

          Show
          khmarbaise Karl Heinz Marbaise added a comment - Andriy Yes you are right about the lightweightness. But having 5000 modules shows the problem using a simple mvn clean and shows the success of your investigations. I have created a patch based on your suggestions. Many thanks for your help and investigation.
          Hide
          khmarbaise Karl Heinz Marbaise added a comment -
          Show
          khmarbaise Karl Heinz Marbaise added a comment - Patch done in 41144e7ecf52e7ec3850f3e78d81f42f505f4af8
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in maven-3.x #1289 (See https://builds.apache.org/job/maven-3.x/1289/)
          MNG-6030 ReactorModelCache do not used effectively after maven version (khmarbaise: rev 41144e7ecf52e7ec3850f3e78d81f42f505f4af8)

          • maven-core/src/main/java/org/apache/maven/project/DefaultProjectBuilder.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in maven-3.x #1289 (See https://builds.apache.org/job/maven-3.x/1289/ ) MNG-6030 ReactorModelCache do not used effectively after maven version (khmarbaise: rev 41144e7ecf52e7ec3850f3e78d81f42f505f4af8) maven-core/src/main/java/org/apache/maven/project/DefaultProjectBuilder.java
          Hide
          stephenc Stephen Connolly added a comment -

          Maven 3.4.0 has been dropped. See this thread for more details.

          This issue will need to be re-scheduled for a Maven release in the (hopefully near) future.

          Show
          stephenc Stephen Connolly added a comment - Maven 3.4.0 has been dropped. See this thread for more details. This issue will need to be re-scheduled for a Maven release in the (hopefully near) future.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build maven-3.x #1508 (See https://builds.apache.org/job/maven-3.x/1508/)
          MNG-6030 ReactorModelCache do not used effectively after maven version (khmarbaise: http://git-wip-us.apache.org/repos/asf/?p=maven.git&a=commit&h=cfb075ac706b25df630f3671f61f8d8313e0f138)

          • (edit) maven-core/src/main/java/org/apache/maven/project/DefaultProjectBuilder.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build maven-3.x #1508 (See https://builds.apache.org/job/maven-3.x/1508/ ) MNG-6030 ReactorModelCache do not used effectively after maven version (khmarbaise: http://git-wip-us.apache.org/repos/asf/?p=maven.git&a=commit&h=cfb075ac706b25df630f3671f61f8d8313e0f138 ) (edit) maven-core/src/main/java/org/apache/maven/project/DefaultProjectBuilder.java

            People

            • Assignee:
              khmarbaise Karl Heinz Marbaise
              Reporter:
              sandriy Andriy
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development