Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13896

Invoke assembly plugin after building jars

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None

      Description

      From what I can tell, all of the hdfs and common jars from their respective lib dirs are missing, excluding hadoop-hdfs-client and hadoop-hdfs-nfs. But there are likely more.

      Steps to reproduce:

      1. ./start-build-env.sh
      2. mvn install -Pdist,src -Dtar -DskipTests -Pnative
      3. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common
      hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps
      4. ls hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs
      hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps

        Activity

        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11076 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11076/)
        HADOOP-13896. Invoke assembly plugin after building jars. (wang: rev 02766b6c224bc163a6fad37f146ade063e7fb422)

        • (edit) hadoop-project-dist/pom.xml
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11076 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11076/ ) HADOOP-13896 . Invoke assembly plugin after building jars. (wang: rev 02766b6c224bc163a6fad37f146ade063e7fb422) (edit) hadoop-project-dist/pom.xml
        Hide
        aw Allen Wittenauer added a comment -

        It was gonna be the next thing on my list. But for a variety of reasons, I'm not going to spend any time on it.

        Show
        aw Allen Wittenauer added a comment - It was gonna be the next thing on my list. But for a variety of reasons, I'm not going to spend any time on it.
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk, thanks again for the review Allen!

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk, thanks again for the review Allen!
        Hide
        andrew.wang Andrew Wang added a comment -

        Yea, I honestly still don't understand the dist / assembly build. I think a lot of the grossness originates from the project split/unsplit, there really should be one place where all of this happens.

        Thanks for the review, I'll commit this shortly.

        Show
        andrew.wang Andrew Wang added a comment - Yea, I honestly still don't understand the dist / assembly build. I think a lot of the grossness originates from the project split/unsplit, there really should be one place where all of this happens. Thanks for the review, I'll commit this shortly.
        Hide
        aw Allen Wittenauer added a comment -

        shrugs

        Seems to be working for me. +1

        At some point, we really need to rework how all of this is laid out. It's pretty terrible. I'd love to use the same trick as we do for tools to only load what we need, but... yeah....

        Show
        aw Allen Wittenauer added a comment - shrugs Seems to be working for me. +1 At some point, we really need to rework how all of this is laid out. It's pretty terrible. I'd love to use the same trick as we do for tools to only load what we need, but... yeah....
        Hide
        andrew.wang Andrew Wang added a comment -

        No tests since this is not something covered by unit tests. Review would be appreciated.

        Side note, I had to rekick the build bot since the first time it failed with a weird error and didn't post a comment:

        https://builds.apache.org/job/PreCommit-HADOOP-Build/11349/

        Show
        andrew.wang Andrew Wang added a comment - No tests since this is not something covered by unit tests. Review would be appreciated. Side note, I had to rekick the build bot since the first time it failed with a weird error and didn't post a comment: https://builds.apache.org/job/PreCommit-HADOOP-Build/11349/
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 12m 21s trunk passed
        +1 compile 0m 8s trunk passed
        +1 mvnsite 0m 11s trunk passed
        +1 mvneclipse 0m 10s trunk passed
        +1 javadoc 0m 9s trunk passed
        +1 mvninstall 0m 7s the patch passed
        +1 compile 0m 6s the patch passed
        +1 javac 0m 6s the patch passed
        +1 mvnsite 0m 8s the patch passed
        +1 mvneclipse 0m 6s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 6s the patch passed
        +1 unit 0m 6s hadoop-project-dist in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        14m 42s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845462/hadoop-13896.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 08d0453b2eca 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a0a2761
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11357/testReport/
        modules C: hadoop-project-dist U: hadoop-project-dist
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11357/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 21s trunk passed +1 compile 0m 8s trunk passed +1 mvnsite 0m 11s trunk passed +1 mvneclipse 0m 10s trunk passed +1 javadoc 0m 9s trunk passed +1 mvninstall 0m 7s the patch passed +1 compile 0m 6s the patch passed +1 javac 0m 6s the patch passed +1 mvnsite 0m 8s the patch passed +1 mvneclipse 0m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 6s the patch passed +1 unit 0m 6s hadoop-project-dist in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 14m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845462/hadoop-13896.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 08d0453b2eca 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0a2761 Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11357/testReport/ modules C: hadoop-project-dist U: hadoop-project-dist Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11357/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        A little more progress debugging this, the src profile is causing the per-component assembly plugin invocation to run before the jars are created. I ran a dist and a dist,src build, and then grepped as follows: egrep "(maven-assembly-plugin|maven-jar-plugin)" dist.out | grep hadoop-common

        dist:

        [INFO] --- maven-jar-plugin:2.5:jar (prepare-jar) @ hadoop-common ---
        [INFO] --- maven-jar-plugin:2.5:test-jar (prepare-test-jar) @ hadoop-common ---
        [INFO] --- maven-assembly-plugin:2.4:single (dist) @ hadoop-common ---
        [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-common ---
        

        dist,src:

        [INFO] --- maven-assembly-plugin:2.4:single (dist) @ hadoop-common ---
        [INFO] --- maven-jar-plugin:2.5:jar (prepare-jar) @ hadoop-common ---
        [INFO] --- maven-jar-plugin:2.5:test-jar (prepare-test-jar) @ hadoop-common ---
        [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-common ---
        

        Here's a patch that just changes the dist assembly plugin in "package" instead of "prepare-package", which fixes the ordering problem. I did a build and diffed the jars in alpha1 against trunk with this patch applied, and only found the changed lib dependencies and the new shaded client and aliyun JARs.

        Show
        andrew.wang Andrew Wang added a comment - A little more progress debugging this, the src profile is causing the per-component assembly plugin invocation to run before the jars are created. I ran a dist and a dist,src build, and then grepped as follows: egrep "(maven-assembly-plugin|maven-jar-plugin)" dist.out | grep hadoop-common dist: [INFO] --- maven-jar-plugin:2.5:jar (prepare-jar) @ hadoop-common --- [INFO] --- maven-jar-plugin:2.5:test-jar (prepare-test-jar) @ hadoop-common --- [INFO] --- maven-assembly-plugin:2.4:single (dist) @ hadoop-common --- [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-common --- dist,src: [INFO] --- maven-assembly-plugin:2.4:single (dist) @ hadoop-common --- [INFO] --- maven-jar-plugin:2.5:jar (prepare-jar) @ hadoop-common --- [INFO] --- maven-jar-plugin:2.5:test-jar (prepare-test-jar) @ hadoop-common --- [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-common --- Here's a patch that just changes the dist assembly plugin in "package" instead of "prepare-package", which fixes the ordering problem. I did a build and diffed the jars in alpha1 against trunk with this patch applied, and only found the changed lib dependencies and the new shaded client and aliyun JARs.
        Hide
        andrew.wang Andrew Wang added a comment -

        I looked at this a little more, and I think I was able to reproduce the issue with mvn clean install -Pdist,src -Dtar -DskipTests. This makes me think it's something to do with the src profile.

        Show
        andrew.wang Andrew Wang added a comment - I looked at this a little more, and I think I was able to reproduce the issue with mvn clean install -Pdist,src -Dtar -DskipTests . This makes me think it's something to do with the src profile.
        Hide
        andrew.wang Andrew Wang added a comment -

        I did a mvn clean package -Pdist -DskipTests -D tar and looked at the resulting tarball and it looks okay:

        -> % ls share/hadoop/common 
        hadoop-common-3.0.0-alpha2-SNAPSHOT.jar        jdiff      webapps
        hadoop-common-3.0.0-alpha2-SNAPSHOT-tests.jar  lib
        hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar           templates
        -> % ls share/hadoop/hdfs  
        hadoop-hdfs-3.0.0-alpha2-SNAPSHOT.jar
        hadoop-hdfs-3.0.0-alpha2-SNAPSHOT-tests.jar
        hadoop-hdfs-client-3.0.0-alpha2-SNAPSHOT-tests.jar
        hadoop-hdfs-native-client-3.0.0-alpha2-SNAPSHOT.jar
        hadoop-hdfs-native-client-3.0.0-alpha2-SNAPSHOT-tests.jar
        hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar
        jdiff
        lib
        templates
        webapps
        
        Show
        andrew.wang Andrew Wang added a comment - I did a mvn clean package -Pdist -DskipTests -D tar and looked at the resulting tarball and it looks okay: -> % ls share/hadoop/common hadoop-common-3.0.0-alpha2-SNAPSHOT.jar jdiff webapps hadoop-common-3.0.0-alpha2-SNAPSHOT-tests.jar lib hadoop-nfs-3.0.0-alpha2-SNAPSHOT.jar templates -> % ls share/hadoop/hdfs hadoop-hdfs-3.0.0-alpha2-SNAPSHOT.jar hadoop-hdfs-3.0.0-alpha2-SNAPSHOT-tests.jar hadoop-hdfs-client-3.0.0-alpha2-SNAPSHOT-tests.jar hadoop-hdfs-native-client-3.0.0-alpha2-SNAPSHOT.jar hadoop-hdfs-native-client-3.0.0-alpha2-SNAPSHOT-tests.jar hadoop-hdfs-nfs-3.0.0-alpha2-SNAPSHOT.jar jdiff lib templates webapps
        Hide
        andrew.wang Andrew Wang added a comment -

        The alpha1 tarball looks okay, so I wonder what changed:

        -> % ls share/hadoop/common 
        hadoop-common-3.0.0-alpha1.jar        hadoop-nfs-3.0.0-alpha1.jar  lib      templates
        hadoop-common-3.0.0-alpha1-tests.jar  jdiff                        sources
        -> % ls share/hadoop/hdfs 
        hadoop-hdfs-3.0.0-alpha1.jar                      jdiff
        hadoop-hdfs-3.0.0-alpha1-tests.jar                lib
        hadoop-hdfs-client-3.0.0-alpha1-tests.jar         sources
        hadoop-hdfs-native-client-3.0.0-alpha1.jar        templates
        hadoop-hdfs-native-client-3.0.0-alpha1-tests.jar  webapps
        hadoop-hdfs-nfs-3.0.0-alpha1.jar
        
        Show
        andrew.wang Andrew Wang added a comment - The alpha1 tarball looks okay, so I wonder what changed: -> % ls share/hadoop/common hadoop-common-3.0.0-alpha1.jar hadoop-nfs-3.0.0-alpha1.jar lib templates hadoop-common-3.0.0-alpha1-tests.jar jdiff sources -> % ls share/hadoop/hdfs hadoop-hdfs-3.0.0-alpha1.jar jdiff hadoop-hdfs-3.0.0-alpha1-tests.jar lib hadoop-hdfs-client-3.0.0-alpha1-tests.jar sources hadoop-hdfs-native-client-3.0.0-alpha1.jar templates hadoop-hdfs-native-client-3.0.0-alpha1-tests.jar webapps hadoop-hdfs-nfs-3.0.0-alpha1.jar
        Hide
        aw Allen Wittenauer added a comment -

        FYI: I have not confirmed if this happens in any other branches.

        Show
        aw Allen Wittenauer added a comment - FYI: I have not confirmed if this happens in any other branches.

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            aw Allen Wittenauer
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development