Uploaded image for project: 'Maven'
  1. Maven
  2. MNG-6843

Parallel build fails due to missing JAR artifacts in compilePath

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.6.3
    • 3.8.2, 4.0.0-alpha-2, 4.0.0
    • Core
    • None
    • - Linux (tested Docker using maven:3-jdk-8 tag): happens most times.
      - Windows 10: happens sometimes.

    Description

      Build of our multi module (57) Java maven project is failing phase when running it as parallel in 4 threads (mvn -T 4 clean install). The failure happens during compilation because packages/classes from compile dependencies cannot be found:

      [main] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project common: Compilation failure: Compilation failure: 
[main] [ERROR] /home/common/src/main/java/com/foo/ZonedDateTimeParser.java:[6,32] package org.apache.commons.lang3 does not exist

      After enabling debug logging (with thread names) I have found out that a compile path of the failing module is empty (besides target/classes):

      When running in 4 threads (-T 4):

      [BuilderThread 2] [DEBUG] (f) compilePath = [/home/common/target/classes]
...
[BuilderThread 2] [DEBUG] Command line options:
[BuilderThread 2] [DEBUG] -d /home/common/target/classes -classpath /home/common/target/classes: -sourcepath /home/common/src/main/java:/home/common/target/generated-sources/annotations: -s /home/common/target/generated-sources/annotations -g -nowarn -target 1.8 -source 1.8 -encoding UTF-8

      When running in a single thread (-T 1):

      [BuilderThread 0] [DEBUG] (f) compilePath = [/home/common/target/classes, /root/.m2/repository/commons-beanutils/commons-beanutils/1.9.3/commons-beanutils-1.9.3.jar, /root/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar, /root/.m2/repository/org/apache/commons/commons-collections4/4.3/commons-collections4-4.3.jar, /root/.m2/repository/org/apache/commons/commons-lang3/3.9/commons-lang3-3.9.jar, /root/.m2/repository/org/jooq/jool-java-8/0.9.14/jool-java-8-0.9.14.jar, /root/.m2/repository/org/slf4j/slf4j-api/1.7.26/slf4j-api-1.7.26.jar, /root/.m2/repository/org/springframework/spring-beans/5.1.8.RELEASE/spring-beans-5.1.8.RELEASE.jar, /root/.m2/repository/org/springframework/spring-core/5.1.8.RELEASE/spring-core-5.1.8.RELEASE.jar, /root/.m2/repository/org/springframework/spring-context/5.1.8.RELEASE/spring-context-5.1.8.RELEASE.jar]
...
[BuilderThread 0] [DEBUG] Command line options:
[BuilderThread 0] [DEBUG] -d /home/common/target/classes -classpath /home/common/target/classes:/root/.m2/repository/commons-beanutils/commons-beanutils/1.9.3/commons-beanutils-1.9.3.jar:/root/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/root/.m2/repository/org/apache/commons/commons-collections4/4.3/commons-collections4-4.3.jar:/root/.m2/repository/org/apache/commons/commons-lang3/3.9/commons-lang3-3.9.jar:/root/.m2/repository/org/jooq/jool-java-8/0.9.14/jool-java-8-0.9.14.jar:/root/.m2/repository/org/slf4j/slf4j-api/1.7.26/slf4j-api-1.7.26.jar:/root/.m2/repository/org/springframework/spring-beans/5.1.8.RELEASE/spring-beans-5.1.8.RELEASE.jar:/root/.m2/repository/org/springframework/spring-core/5.1.8.RELEASE/spring-core-5.1.8.RELEASE.jar:/root/.m2/repository/org/springframework/spring-context/5.1.8.RELEASE/spring-context-5.1.8.RELEASE.jar: -sourcepath /home/common/src/main/java:/home/common/target/generated-sources/annotations: -s /home/common/target/generated-sources/annotations -g -nowarn -target 1.8 -source 1.8 -encoding UTF-8

      After adding custom log messages I have found out that the root cause is that org.apache.maven.project.MavenProject.setArtifactFilter() is called with null artifactFilter parameter. The call happens for the failing module from a thread that is building another module. The call stack is:

      "BuilderThread 0@2513" prio=5 tid=0xe nid=NA runnable
 java.lang.Thread.State: RUNNABLE
 at org.apache.maven.project.MavenProject.setArtifactFilter(MavenProject.java:1437)
 at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:279)
 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:202)
 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
 at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
 at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:190)
 at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:186)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

      The MavenProject objects are shared among all build threads via MavenSession - member org.apache.maven.execution.MavenSession#projects.
      As all the MavenSession objects are created by cloning an initial MavenSession object, I have added cloning of projects into the MavenSession clone() method. With this change the build succeeded:

      @Override
public MavenSession clone()
{
    try
    {
        MavenSession thisClone = (MavenSession) super.clone();
        thisClone.setProjects(getProjectsClone());
        return thisClone;
    }
    catch ( CloneNotSupportedException e )
    {
        throw new RuntimeException( "Bug", e );
    }
}

private synchronized List<MavenProject> getProjectsClone()
{
    if (projects == null) {
        return null;
    } else {
        if (projects.isEmpty()) {
            return Collections.emptyList();
        } else {
            List<MavenProject> clonedProjects = new ArrayList<>(projects.size());
            for (MavenProject project : projects) {
                clonedProjects.add(project.clone());
            }            return clonedProjects;
        }
    }
}

      Attachments

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. MNG-7285 [Regression] MavenProject.getArtifacts() not returning correct value across multiple threads

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MNG-7312 Revert ThreadLocal approach from MNG-6843 and MNG-7251

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MNG-7220 [REGRESSION] test-classpath incorrectly resolved

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MNG-4526 [regression] Shade plugin includes too much (relative to 2.2.1)

          • Major - Major loss of function.
          • Closed
          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MNG-7335 [Regression] Parallel build fails due to missing JAR artifacts in compilePath

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MNG-7157 Provide a better design for obtaining artifacts from a MavenProject inside a mojo

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MNG-7212 Prefer use of InheritableThreadLocal to fix possible issue when mojo starts a thread and tries to access some artifacts

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #310

          Web Link GitHub Pull Request #413

          Web Link GitHub Pull Request #482

          Activity

            People

              rfscholte Robert Scholte
              stepan.hrbacek@gmail.com Stepan Hrbacek
              Votes:
              7 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m