[SPARK-26095] make-distribution.sh is hanging in jenkins

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.6, 3.0.0
• Fix Version/s: 2.4.6, 3.0.0
• Component/s: Build
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 16/Nov/18 21:49
• Updated: 29/May/20 21:00
• Resolved: 16/Nov/18 23:57

Description

See https://github.com/apache/spark/pull/23017 for further discussion.

maven seems to get stuck here:

"BuilderThread 5" #80 prio=5 os_prio=0 tid=0x00007f16b8500000 nid=0x7bcf runnable [0x00007f16882fd000]
   java.lang.Thread.State: RUNNABLE
        at org.jdom2.Element.isAncestor(Element.java:1052)
        at org.jdom2.ContentList.checkPreConditions(ContentList.java:222)
        at org.jdom2.ContentList.add(ContentList.java:244)
        at org.jdom2.Element.addContent(Element.java:950)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.insertAtPreferredLocation(MavenJDOMWriter.java:292)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.iterateExclusion(MavenJDOMWriter.java:488)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.updateDependency(MavenJDOMWriter.java:1335)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.iterateDependency(MavenJDOMWriter.java:386)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.updateModel(MavenJDOMWriter.java:1623)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.write(MavenJDOMWriter.java:2156)
        at org.apache.maven.plugins.shade.pom.PomWriter.write(PomWriter.java:75)
        at org.apache.maven.plugins.shade.mojo.ShadeMojo.rewriteDependencyReducedPomIfWeHaveReduction(ShadeMojo.java:1049)
        at org.apache.maven.plugins.shade.mojo.ShadeMojo.createDependencyReducedPom(ShadeMojo.java:978)
        at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:538)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
        at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:200)
        at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:196)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

And in fact I see a bunch of threads stuck there. Trying a few different things.

Issue Links

• is depended upon by: SPARK-31541 Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging)
• links to: [Github] Pull Request #23061 (vanzin)