Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12761

incremental maven build is not really incremental

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For any version that uses v.3.1 of the maven-compiler-plugin, the incremental maven build is basically broken. For most of the modules, an incremental build (mvn install -DskipTests on an already built directory for example) rebuilds the whole module again:

      [INFO] ------------------------------------------------------------------------
      [INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
      [INFO] ------------------------------------------------------------------------
      [INFO] 
      [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-common ---
      [INFO] Executing tasks
      
      main:
      [INFO] Executed tasks
      [INFO] 
      [INFO] --- hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) @ hadoop-common ---
      [INFO] No changes detected in protoc files, skipping generation.
      [INFO] 
      [INFO] --- hadoop-maven-plugins:3.0.0-SNAPSHOT:version-info (version-info) @ hadoop-common ---
      [WARNING] [svn, info] failed with error code 1
      [INFO] SCM: GIT
      [INFO] Computed MD5: c8e92ce138fcd723204649e4d7c6ddd
      [INFO] 
      [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-common ---
      [INFO] Using 'UTF-8' encoding to copy filtered resources.
      [INFO] Copying 7 resources
      [INFO] Copying 1 resource
      [INFO] 
      [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-common ---
      [INFO] Changes detected - recompiling the module!
      [INFO] Compiling 871 source files to /Users/foo/hadoop/hadoop-common-project/hadoop-common/target/classes
      

      It turns out that the incremental build feature of the maven compiler plugin is basically broken at v3.1 (see http://stackoverflow.com/questions/17944108/maven-compiler-plugin-always-detecting-a-set-of-sources-as-stale and MCOMPILER-209). Ironically, this can be fixed by turning off the "incremental build" configuration of the plugin.

        Activity

        Hide
        sjlee0 Sangjin Lee added a comment -

        Posted patch v.1.

        Show
        sjlee0 Sangjin Lee added a comment - Posted patch v.1.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 7s Maven dependency ordering for branch
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 8s trunk passed with JDK v1.8.0_66
        +1 compile 0m 8s trunk passed with JDK v1.7.0_91
        +1 mvnsite 0m 13s trunk passed
        +1 mvneclipse 0m 9s trunk passed
        +1 javadoc 0m 9s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 9s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 6s Maven dependency ordering for patch
        +1 mvninstall 0m 8s the patch passed
        +1 compile 0m 5s the patch passed with JDK v1.8.0_66
        +1 javac 0m 5s the patch passed
        +1 compile 0m 7s the patch passed with JDK v1.7.0_91
        +1 javac 0m 7s the patch passed
        +1 mvnsite 0m 9s the patch passed
        +1 mvneclipse 0m 7s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 6s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 7s the patch passed with JDK v1.7.0_91
        +1 unit 0m 6s hadoop-project in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 6s hadoop-project in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 16s Patch does not generate ASF License warnings.
        10m 4s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785899/HADOOP-12761.01.patch
        JIRA Issue HADOOP-12761
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 36ecb9613427 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ccbba4a
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8514/testReport/
        modules C: hadoop-project U: hadoop-project
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8514/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 0m 8s trunk passed with JDK v1.8.0_66 +1 compile 0m 8s trunk passed with JDK v1.7.0_91 +1 mvnsite 0m 13s trunk passed +1 mvneclipse 0m 9s trunk passed +1 javadoc 0m 9s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 9s trunk passed with JDK v1.7.0_91 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 0m 8s the patch passed +1 compile 0m 5s the patch passed with JDK v1.8.0_66 +1 javac 0m 5s the patch passed +1 compile 0m 7s the patch passed with JDK v1.7.0_91 +1 javac 0m 7s the patch passed +1 mvnsite 0m 9s the patch passed +1 mvneclipse 0m 7s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 7s the patch passed with JDK v1.7.0_91 +1 unit 0m 6s hadoop-project in the patch passed with JDK v1.8.0_66. +1 unit 0m 6s hadoop-project in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 10m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785899/HADOOP-12761.01.patch JIRA Issue HADOOP-12761 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 36ecb9613427 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ccbba4a Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8514/testReport/ modules C: hadoop-project U: hadoop-project Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8514/console This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        +1 Works for me!

        Show
        kihwal Kihwal Lee added a comment - +1 Works for me!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9238 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9238/)
        HADOOP-12761. incremental maven build is not really incremental (sjlee) (sjlee: rev 4dc0a3949386ce2961356143a5a843dd537829dc)

        • hadoop-project/pom.xml
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9238 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9238/ ) HADOOP-12761 . incremental maven build is not really incremental (sjlee) (sjlee: rev 4dc0a3949386ce2961356143a5a843dd537829dc) hadoop-project/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        sjlee0 Sangjin Lee added a comment -

        Committed it to trunk, branch-2, branch-2.8, and branch-2.7. Thanks Kihwal Lee for your +1!

        Show
        sjlee0 Sangjin Lee added a comment - Committed it to trunk, branch-2, branch-2.8, and branch-2.7. Thanks Kihwal Lee for your +1!
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            sjlee0 Sangjin Lee
            Reporter:
            sjlee0 Sangjin Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development