Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13689

Do not attach javadoc and sources jars during non-dist build

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None

      Description

      Looking at maven output when running with "-Pdist", the source plugin "test-jar" and "jar" goals are invoked twice. This is because it's turned on by both the dist profile and on by default.

      Outside of the release context, it's not that important to have javadoc and source JARs, so I think we can turn it off by default.

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Simple patch attached. The maven source plugin also says to prefer the "no-fork" goals, which is what the dist profile does in hadoop-project/pom.xml.

        Show
        andrew.wang Andrew Wang added a comment - Simple patch attached. The maven source plugin also says to prefer the "no-fork" goals, which is what the dist profile does in hadoop-project/pom.xml.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 5s trunk passed
        +1 compile 0m 8s trunk passed
        +1 mvnsite 0m 11s trunk passed
        +1 mvneclipse 0m 11s trunk passed
        +1 javadoc 0m 8s trunk passed
        +1 mvninstall 0m 7s the patch passed
        +1 compile 0m 6s the patch passed
        +1 javac 0m 6s the patch passed
        +1 mvnsite 0m 8s the patch passed
        +1 mvneclipse 0m 6s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 6s the patch passed
        +1 unit 0m 6s hadoop-project-dist in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        10m 26s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13689
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831896/HADOOP-13689.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux e419eaa2e809 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2023254
        Default Java 1.8.0_101
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10685/testReport/
        modules C: hadoop-project-dist U: hadoop-project-dist
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10685/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 5s trunk passed +1 compile 0m 8s trunk passed +1 mvnsite 0m 11s trunk passed +1 mvneclipse 0m 11s trunk passed +1 javadoc 0m 8s trunk passed +1 mvninstall 0m 7s the patch passed +1 compile 0m 6s the patch passed +1 javac 0m 6s the patch passed +1 mvnsite 0m 8s the patch passed +1 mvneclipse 0m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 6s the patch passed +1 unit 0m 6s hadoop-project-dist in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 10m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13689 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831896/HADOOP-13689.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux e419eaa2e809 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2023254 Default Java 1.8.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10685/testReport/ modules C: hadoop-project-dist U: hadoop-project-dist Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10685/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Nice find Andrew. Tested this locally. +1

        Show
        xiaochen Xiao Chen added a comment - Nice find Andrew. Tested this locally. +1
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the review Xiao, committed to trunk, branch-2, branch-2.8.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for the review Xiao, committed to trunk, branch-2, branch-2.8.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10562 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10562/)
        HADOOP-13689. Do not attach javadoc and sources jars during non-dist (wang: rev bf372173d0f7cb97b62556cbd199a075254b96e6)

        • (edit) hadoop-project-dist/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10562 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10562/ ) HADOOP-13689 . Do not attach javadoc and sources jars during non-dist (wang: rev bf372173d0f7cb97b62556cbd199a075254b96e6) (edit) hadoop-project-dist/pom.xml
        Hide
        musiced eddie luo added a comment -

        Hi, it seems that this change would prevent the distributing hadoop-common.jar into

        {BUILD_DIR}

        /share/hadoop/common/hadoop-common.jar.
        It will cause build failure when i am using most current BIGTOP to install hadoop 2.8.0.

        Show
        musiced eddie luo added a comment - Hi, it seems that this change would prevent the distributing hadoop-common.jar into {BUILD_DIR} /share/hadoop/common/hadoop-common.jar. It will cause build failure when i am using most current BIGTOP to install hadoop 2.8.0.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Eddie, can you provide your maven invocation? Are you doing a dist build?

        Show
        andrew.wang Andrew Wang added a comment - Hi Eddie, can you provide your maven invocation? Are you doing a dist build?
        Hide
        jonathak Jonathan Kelly added a comment -

        Andrew Wang, Apache Bigtop invokes it here and includes -Pdist.

        Show
        jonathak Jonathan Kelly added a comment - Andrew Wang , Apache Bigtop invokes it here and includes -Pdist.
        Hide
        andrew.wang Andrew Wang added a comment -

        I ran "mvn package -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip", extracted the resulting tarball, and found this:

        -> % find . -name "hadoop-common*.jar"
        ./hadoop-2.8.1-SNAPSHOT/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/hadoop-common-2.8.1-SNAPSHOT.jar
        ./hadoop-2.8.1-SNAPSHOT/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/hadoop-common-2.8.1-SNAPSHOT.jar
        ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/hadoop-common-2.8.1-SNAPSHOT.jar
        ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/sources/hadoop-common-2.8.1-SNAPSHOT-test-sources.jar
        ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/sources/hadoop-common-2.8.1-SNAPSHOT-sources.jar
        ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/hadoop-common-2.8.1-SNAPSHOT-tests.jar
        

        This looks right to me. We have the source jars even too.

        Is there a JIRA with more details? Have you tried reverting this pom change locally in bigtop to verify your error is caused by this change? There's a lot that went into 2.8.0, chances are things have moved around.

        Show
        andrew.wang Andrew Wang added a comment - I ran "mvn package -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip", extracted the resulting tarball, and found this: -> % find . -name "hadoop-common*.jar" ./hadoop-2.8.1-SNAPSHOT/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/hadoop-common-2.8.1-SNAPSHOT.jar ./hadoop-2.8.1-SNAPSHOT/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/hadoop-common-2.8.1-SNAPSHOT.jar ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/hadoop-common-2.8.1-SNAPSHOT.jar ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/sources/hadoop-common-2.8.1-SNAPSHOT-test-sources.jar ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/sources/hadoop-common-2.8.1-SNAPSHOT-sources.jar ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/hadoop-common-2.8.1-SNAPSHOT-tests.jar This looks right to me. We have the source jars even too. Is there a JIRA with more details? Have you tried reverting this pom change locally in bigtop to verify your error is caused by this change? There's a lot that went into 2.8.0, chances are things have moved around.
        Hide
        musiced eddie luo added a comment -

        I didn't open a JIRA for this issue.
        Yes, i fixed this error by reverting this change.
        Synced with hadoop 2.8 about three weeks ago. But found this problem around two months ago,
        The tip on my branch is
        Author: Wei-Chiu Chuang <weichiu@apache.org>
        Date: Wed Feb 15 02:50:35 2017 -0800

        HDFS-11333. Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang.

        Show
        musiced eddie luo added a comment - I didn't open a JIRA for this issue. Yes, i fixed this error by reverting this change. Synced with hadoop 2.8 about three weeks ago. But found this problem around two months ago, The tip on my branch is Author: Wei-Chiu Chuang <weichiu@apache.org> Date: Wed Feb 15 02:50:35 2017 -0800 HDFS-11333 . Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang.
        Hide
        andrew.wang Andrew Wang added a comment -

        Best I can tell from my test, the hadoop jars are still in the tarball. If you could isolate the issue outside of Bigtop, like the maven command I ran in my previous comment, that would be really helpful in diagnosing this.

        Show
        andrew.wang Andrew Wang added a comment - Best I can tell from my test, the hadoop jars are still in the tarball. If you could isolate the issue outside of Bigtop, like the maven command I ran in my previous comment, that would be really helpful in diagnosing this.
        Hide
        musiced eddie luo added a comment -

        Hello:
        If you run "mvn clean install -DskipTests -Pdist -Dtar -Psrc -Dmaven.javadoc.skip"
        then hadoop-common jar is not under
        ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/

        Show
        musiced eddie luo added a comment - Hello: If you run "mvn clean install -DskipTests -Pdist -Dtar -Psrc -Dmaven.javadoc.skip" then hadoop-common jar is not under ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/
        Hide
        musiced eddie luo added a comment -

        I think is expected behavior of maven since we removed
        diff --git a/hadoop-project-dist/pom.xml b/hadoop-project-dist/pom.xml
        + <executions>
        + <execution>
        + <phase>prepare-package</phase>
        + <goals>
        + <goal>jar</goal>
        + <goal>test-jar</goal>
        + </goals>
        + </execution>
        + </executions>
        + <configuration>
        + <attach>true</attach>
        + </configuration>
        + </plugin>
        + <plugin>
        then when we run -Pdist -Psrc will not put the hadoop-common and test jar into ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/
        So we are good. Thanks!

        Show
        musiced eddie luo added a comment - I think is expected behavior of maven since we removed diff --git a/hadoop-project-dist/pom.xml b/hadoop-project-dist/pom.xml + <executions> + <execution> + <phase>prepare-package</phase> + <goals> + <goal>jar</goal> + <goal>test-jar</goal> + </goals> + </execution> + </executions> + <configuration> + <attach>true</attach> + </configuration> + </plugin> + <plugin> then when we run -Pdist -Psrc will not put the hadoop-common and test jar into ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/ So we are good. Thanks!

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development