Hadoop Common
  1. Hadoop Common
  2. HADOOP-8066

The full docs build intermittently fails

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0
    • Fix Version/s: 0.23.2
    • Component/s: build
    • Labels:
      None
    1. HADOOP-8066.mk2.patch.txt
      0.5 kB
      Andrew Bayer
    2. HADOOP-8066.patch.txt
      1 kB
      Andrew Bayer

      Activity

      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk #993 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/993/)
      HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250)

      Result = SUCCESS
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #993 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/993/ ) HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/199/)
      Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251)

      Result = FAILURE
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/199/ ) Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk #958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/958/)
      HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250)

      Result = SUCCESS
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/958/ ) HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-0.23-Build #171 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/171/)
      Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251)

      Result = SUCCESS
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #171 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/171/ ) Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-0.23-Commit #565 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/565/)
      Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251)

      Result = ABORTED
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #565 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/565/ ) Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #1750 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1750/)
      HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250)

      Result = ABORTED
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1750 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1750/ ) HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-0.23-Commit #562 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/562/)
      Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251)

      Result = SUCCESS
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #562 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/562/ ) Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-trunk-Commit #1738 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1738/)
      HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250)

      Result = SUCCESS
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1738 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1738/ ) HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-0.23-Commit #549 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/549/)
      Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251)

      Result = SUCCESS
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #549 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/549/ ) Merge -r 1245249:1245250 from trunk to branch. FIXES: HADOOP-8066 (Revision 1245251) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245251 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk-Commit #1812 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1812/)
      HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250)

      Result = SUCCESS
      tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-project/pom.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1812 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1812/ ) HADOOP-8066 The full docs build intermittently fails (abayer via tucu) (Revision 1245250) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245250 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
      Hide
      Alejandro Abdelnur added a comment -

      Thanks Andrew. Committed to trunk and branch-0.23

      Show
      Alejandro Abdelnur added a comment - Thanks Andrew. Committed to trunk and branch-0.23
      Hide
      Alejandro Abdelnur added a comment -

      +1 tested doing a full build and generating site.

      Show
      Alejandro Abdelnur added a comment - +1 tested doing a full build and generating site.
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12514888/HADOOP-8066.mk2.patch.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      -1 patch. The patch command could not apply the patch.

      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/609//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514888/HADOOP-8066.mk2.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/609//console This message is automatically generated.
      Hide
      Andrew Bayer added a comment -

      The new patch, btw, is instead of the original.

      Show
      Andrew Bayer added a comment - The new patch, btw, is instead of the original.
      Hide
      Andrew Bayer added a comment -

      New patch - rather than messing with phases, I'm instead adding a declaration of the findbugs plugin at the top of the plugins list in hadoop-project/pom.xml, so that it'll run before any other plugin in a given phase, so long as that plugin is declared for the first time in hadoop-project or anything inheriting from hadoop-project. Seems to do the trick, and doesn't result in executing findbugs normally, since no execution is specified.

      Show
      Andrew Bayer added a comment - New patch - rather than messing with phases, I'm instead adding a declaration of the findbugs plugin at the top of the plugins list in hadoop-project/pom.xml, so that it'll run before any other plugin in a given phase, so long as that plugin is declared for the first time in hadoop-project or anything inheriting from hadoop-project. Seems to do the trick, and doesn't result in executing findbugs normally, since no execution is specified.
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12514692/HADOOP-8066.patch.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      -1 patch. The patch command could not apply the patch.

      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/601//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514692/HADOOP-8066.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/601//console This message is automatically generated.
      Hide
      Andrew Bayer added a comment -

      This should work for now - it's not ideal, but it guarantees the steps are run in the right order.

      Show
      Andrew Bayer added a comment - This should work for now - it's not ideal, but it guarantees the steps are run in the right order.
      Hide
      Todd Lipcon added a comment -

      This is a regression, right? Any chance we could revert the commit that introduced it while we figure out the solution? Or introduce a workaround even if it's temporary and slows the build? It's bad to not get the nightly test results anymore.

      Show
      Todd Lipcon added a comment - This is a regression, right? Any chance we could revert the commit that introduced it while we figure out the solution? Or introduce a workaround even if it's temporary and slows the build? It's bad to not get the nightly test results anymore.
      Hide
      Andrew Bayer added a comment -

      I'm not sure if that's the right way to go - how often are those going to be called? If regularly, profile+overloading seems the right way to go over having to specify a bunch of goals, even with helper scripts.

      Show
      Andrew Bayer added a comment - I'm not sure if that's the right way to go - how often are those going to be called? If regularly, profile+overloading seems the right way to go over having to specify a bunch of goals, even with helper scripts.
      Hide
      Alejandro Abdelnur added a comment -

      Actually, I would love to remove all this overloading of phases with plugins for reporting/javadocs/sitegen. It makes the POMs much more complicated and it runs tons of additional stuff in different phase.

      Maybe invoking those plugin goals directly (and have a helper script so you don't need to type 10 goals)?

      Show
      Alejandro Abdelnur added a comment - Actually, I would love to remove all this overloading of phases with plugins for reporting/javadocs/sitegen. It makes the POMs much more complicated and it runs tons of additional stuff in different phase. Maybe invoking those plugin goals directly (and have a helper script so you don't need to type 10 goals)?
      Hide
      Andrew Bayer added a comment -

      Yeah, that's what seems to be the case. Given that, I'm not sure we have a real alternative to juggling the phase for at least the findbugs call.

      Show
      Andrew Bayer added a comment - Yeah, that's what seems to be the case. Given that, I'm not sure we have a real alternative to juggling the phase for at least the findbugs call.
      Hide
      Alejandro Abdelnur added a comment -

      Per my understanding, plugins wired explicitly are executed in their order of appearance in the POM. The tricky thing is, and this may be what you are seeing, when you having profiles that add executions to plugins wired in the main section of the POM; in this case, the order of execution is determined by the position of the plugin in the main section of the POM.

      Show
      Alejandro Abdelnur added a comment - Per my understanding, plugins wired explicitly are executed in their order of appearance in the POM. The tricky thing is, and this may be what you are seeing, when you having profiles that add executions to plugins wired in the main section of the POM; in this case, the order of execution is determined by the position of the plugin in the main section of the POM.
      Hide
      Andrew Bayer added a comment -

      Also, side note - https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-0.23-Build/ currently calls "mvn clean install package (etc)" - the package there is redundant, since package gets run as part of install, and so everything up through packages gets re-run a second time for no benefit. Ends up adding like 2-3 minutes to the build.

      Show
      Andrew Bayer added a comment - Also, side note - https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-0.23-Build/ currently calls "mvn clean install package (etc)" - the package there is redundant, since package gets run as part of install, and so everything up through packages gets re-run a second time for no benefit. Ends up adding like 2-3 minutes to the build.
      Hide
      Andrew Bayer added a comment -

      This is due to annoying more-or-less non-deterministic Maven plugin execution order - in hadoop-project-dist/pom.xml (the parent for the failing modules), in the docs profile, there are a bunch of plugins running in the "prepare-package" phase, and based on some factors that are not at all obvious, Maven decides the order they get run in. For whatever reason, the findbugs execution isn't happening before the antrun execution, and so the antrun is failing (since it needs the findbugs execution to have already run).

      What I've done in a local test is switched the phase for the docs profile plugin executions other than antrun from "prepare-package" to "process-test-classes", which will have run by the time "prepare-package" is reached. That seems to do the trick.

      Show
      Andrew Bayer added a comment - This is due to annoying more-or-less non-deterministic Maven plugin execution order - in hadoop-project-dist/pom.xml (the parent for the failing modules), in the docs profile, there are a bunch of plugins running in the "prepare-package" phase, and based on some factors that are not at all obvious, Maven decides the order they get run in. For whatever reason, the findbugs execution isn't happening before the antrun execution, and so the antrun is failing (since it needs the findbugs execution to have already run). What I've done in a local test is switched the phase for the docs profile plugin executions other than antrun from "prepare-package" to "process-test-classes", which will have run by the time "prepare-package" is reached. That seems to do the trick.

        People

        • Assignee:
          Andrew Bayer
          Reporter:
          Aaron T. Myers
        • Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development