Hadoop Common
  1. Hadoop Common
  2. HADOOP-8535

Cut hadoop build times in half (upgrade maven-compiler-plugin to 2.5.1)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: build
    • Labels:
      None

      Description

      starting with 2.4.0 maven-compiler-plugin started caching the class loader for the javac compiler. This patch upgrades the compiler to 2.5.1. On my box, build times are reduced from 5 minutes to 2 minutes.

      1. HADOOP-8535.patch
        0.4 kB
        Jonathan Eagles

        Issue Links

          Activity

          Hide
          Jonathan Eagles added a comment -

          I verified the build builds, installs, and runs using a simple wordcount test.

          Show
          Jonathan Eagles added a comment - I verified the build builds, installs, and runs using a simple wordcount test.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12533669/HADOOP-8535.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1144//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1144//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533669/HADOOP-8535.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1144//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1144//console This message is automatically generated.
          Hide
          Jonathan Eagles added a comment -

          http://jira.codehaus.org/browse/PLXCOMP-202 is the JIRA that added the JavacCompiler classloader cache.

          The build command I executed was

          mvn clean install -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip

          which was performed with an empty .m2 cache (nexus enabled) and a clean git repo at the top level.

          Without Path
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 5:06.711s
          [INFO] Finished at: Wed Jun 27 13:36:54 CDT 2012
          [INFO] Final Memory: 334M/455M
          [INFO] ------------------------------------------------------------------------

          With Patch

          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2:15.486s
          [INFO] Finished at: Wed Jun 27 13:29:18 CDT 2012
          [INFO] Final Memory: 70M/272M
          [INFO] ------------------------------------------------------------------------

          Show
          Jonathan Eagles added a comment - http://jira.codehaus.org/browse/PLXCOMP-202 is the JIRA that added the JavacCompiler classloader cache. The build command I executed was mvn clean install -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip which was performed with an empty .m2 cache (nexus enabled) and a clean git repo at the top level. Without Path [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5:06.711s [INFO] Finished at: Wed Jun 27 13:36:54 CDT 2012 [INFO] Final Memory: 334M/455M [INFO] ------------------------------------------------------------------------ With Patch [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:15.486s [INFO] Finished at: Wed Jun 27 13:29:18 CDT 2012 [INFO] Final Memory: 70M/272M [INFO] ------------------------------------------------------------------------
          Hide
          Robert Joseph Evans added a comment -

          +1 I built with the patch it seems to work just fine and I am in favor of speeding up the builds. Feel free to check this in. Thanks for this Jon.

          Show
          Robert Joseph Evans added a comment - +1 I built with the patch it seems to work just fine and I am in favor of speeding up the builds. Feel free to check this in. Thanks for this Jon.
          Hide
          Robert Joseph Evans added a comment -

          I just turned around and saw that Jon was out today, so I will check it in instead .

          Show
          Robert Joseph Evans added a comment - I just turned around and saw that Jon was out today, so I will check it in instead .
          Hide
          Robert Joseph Evans added a comment -

          Thanks again Jon. I put this into trunk, branch-2 and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks again Jon. I put this into trunk, branch-2 and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2470 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2470/)
          HADOOP-8535. Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2470 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2470/ ) HADOOP-8535 . Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2401 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2401/)
          HADOOP-8535. Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2401 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2401/ ) HADOOP-8535 . Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2420 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2420/)
          HADOOP-8535. Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2420 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2420/ ) HADOOP-8535 . Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1091/)
          HADOOP-8535. Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1091/ ) HADOOP-8535 . Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1124 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1124/)
          HADOOP-8535. Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1124 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1124/ ) HADOOP-8535 . Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355019) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #301 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/301/)
          svn merge -c 1355019 FIXES: HADOOP-8535. Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355021)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355021
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #301 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/301/ ) svn merge -c 1355019 FIXES: HADOOP-8535 . Cut hadoop build times in half (Job Eagles via bobby) (Revision 1355021) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355021 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml

            People

            • Assignee:
              Jonathan Eagles
              Reporter:
              Jonathan Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development