Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None

      Description

      Found some build issues while doing some test runs with the create-release.sh script.

      1. hadoop-13996.001.patch
        1 kB
        Andrew Wang
      2. hadoop-13996.002.patch
        1 kB
        Andrew Wang

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Fixed two issues:

        • Apparently you can't run mvn clean without the hadoop maven plugins, so install those first. It was trying to download a SNAPSHOT which was failing.
        • The sign plugin expects there to be a JAR generated by every project. Since we disabled generating the dummy hadoop-dist jar, also disable the sign plugin.
        Show
        andrew.wang Andrew Wang added a comment - Fixed two issues: Apparently you can't run mvn clean without the hadoop maven plugins, so install those first. It was trying to download a SNAPSHOT which was failing. The sign plugin expects there to be a JAR generated by every project. Since we disabled generating the dummy hadoop-dist jar, also disable the sign plugin.
        Hide
        andrew.wang Andrew Wang added a comment -

        Allen Wittenauer perchance you can do a quick review?

        Show
        andrew.wang Andrew Wang added a comment - Allen Wittenauer perchance you can do a quick review?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 58s trunk passed
        +1 compile 0m 12s trunk passed
        +1 mvnsite 0m 13s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 javadoc 0m 11s trunk passed
        +1 mvninstall 0m 11s the patch passed
        +1 compile 0m 9s the patch passed
        +1 javac 0m 9s the patch passed
        +1 mvnsite 0m 11s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 shellcheck 0m 12s The patch generated 0 new + 106 unchanged - 1 fixed = 106 total (was 107)
        +1 shelldocs 0m 9s There were no new shelldocs issues.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 javadoc 0m 7s the patch passed
        +1 unit 0m 9s hadoop-dist in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        17m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13996
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847908/hadoop-13996.001.patch
        Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 42f6aedbdb5b 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4d1f3d9
        Default Java 1.8.0_111
        shellcheck v0.4.5
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11450/testReport/
        modules C: hadoop-dist U: hadoop-dist
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11450/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 58s trunk passed +1 compile 0m 12s trunk passed +1 mvnsite 0m 13s trunk passed +1 mvneclipse 0m 14s trunk passed +1 javadoc 0m 11s trunk passed +1 mvninstall 0m 11s the patch passed +1 compile 0m 9s the patch passed +1 javac 0m 9s the patch passed +1 mvnsite 0m 11s the patch passed +1 mvneclipse 0m 11s the patch passed +1 shellcheck 0m 12s The patch generated 0 new + 106 unchanged - 1 fixed = 106 total (was 107) +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 0m 7s the patch passed +1 unit 0m 9s hadoop-dist in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 17m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13996 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847908/hadoop-13996.001.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml uname Linux 42f6aedbdb5b 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4d1f3d9 Default Java 1.8.0_111 shellcheck v0.4.5 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11450/testReport/ modules C: hadoop-dist U: hadoop-dist Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11450/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        aw Allen Wittenauer added a comment -

        I'm having trouble getting a full build to complete due to the gpg-agent timing out while running it in a Docker container on my Mac. Now that the build takes significant longer (YARN UI, shading, and probably some other big, time consumers have been added), we probably need to bump up the TTL too.

        I'll play with it some more.

        Show
        aw Allen Wittenauer added a comment - I'm having trouble getting a full build to complete due to the gpg-agent timing out while running it in a Docker container on my Mac. Now that the build takes significant longer (YARN UI, shading, and probably some other big, time consumers have been added), we probably need to bump up the TTL too. I'll play with it some more.
        Hide
        aw Allen Wittenauer added a comment -

        FWIW, I'm also seeing:

        • hadoop-client-minicluster throwing some serious warnings.
        • failure during shading of hadoop-client-runtime while building the javadocs

        This is with "create-release --asfrelease --docker".

        At this point, I'm really concerned about the stability of the tree. I've been debating adding a create-release run in the qbt build and it's looking like a better idea more and more.

        Show
        aw Allen Wittenauer added a comment - FWIW, I'm also seeing: hadoop-client-minicluster throwing some serious warnings. failure during shading of hadoop-client-runtime while building the javadocs This is with "create-release --asfrelease --docker". At this point, I'm really concerned about the stability of the tree. I've been debating adding a create-release run in the qbt build and it's looking like a better idea more and more.
        Hide
        andrew.wang Andrew Wang added a comment -

        FWIW I am able to build a release with these changes on my computer, so I'd like to get them in so we can branch and kick off an RC by next week.

        If you want to JIRA these other issues, I promise to review promptly.

        Show
        andrew.wang Andrew Wang added a comment - FWIW I am able to build a release with these changes on my computer, so I'd like to get them in so we can branch and kick off an RC by next week. If you want to JIRA these other issues, I promise to review promptly.
        Hide
        aw Allen Wittenauer added a comment -

        the TTL increase should be part of this patch given that create-release is the one that sets it.

        Show
        aw Allen Wittenauer added a comment - the TTL increase should be part of this patch given that create-release is the one that sets it.
        Hide
        andrew.wang Andrew Wang added a comment -

        Sure, here's a new patch which also doubles the gpg agent TTL.

        Show
        andrew.wang Andrew Wang added a comment - Sure, here's a new patch which also doubles the gpg agent TTL.
        Hide
        aw Allen Wittenauer added a comment -

        OK, I lazily doubled the ttl and that fixed the signing issues. The other stuff can certainly be done elsewhere.

        Show
        aw Allen Wittenauer added a comment - OK, I lazily doubled the ttl and that fixed the signing issues. The other stuff can certainly be done elsewhere.
        Hide
        aw Allen Wittenauer added a comment -

        +1

        Show
        aw Allen Wittenauer added a comment - +1
        Hide
        aw Allen Wittenauer added a comment -

        (FWIW, I'm curious as to what changed that cause the clean to not install the maven plugin prior to executing, but not enough to dig it out.)

        Show
        aw Allen Wittenauer added a comment - (FWIW, I'm curious as to what changed that cause the clean to not install the maven plugin prior to executing, but not enough to dig it out.)
        Hide
        andrew.wang Andrew Wang added a comment -

        I think it might have been related to the SNAPSHOT builds failing when the shading patch first went in. Regardless, it seems like we should follow the instructions in our own BUILDING.txt when building a release.

        Thanks for the review, I'll commit this shortly.

        Show
        andrew.wang Andrew Wang added a comment - I think it might have been related to the SNAPSHOT builds failing when the shading patch first went in. Regardless, it seems like we should follow the instructions in our own BUILDING.txt when building a release. Thanks for the review, I'll commit this shortly.
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk, thanks for the reviews Allen!

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for the reviews Allen!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11137 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11137/)
        HADOOP-13996. Fix some release build issues. (wang: rev 2bf4c6e2d622570af43f50a5f3738dfe0004b55b)

        • (edit) hadoop-dist/pom.xml
        • (edit) dev-support/bin/create-release
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11137 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11137/ ) HADOOP-13996 . Fix some release build issues. (wang: rev 2bf4c6e2d622570af43f50a5f3738dfe0004b55b) (edit) hadoop-dist/pom.xml (edit) dev-support/bin/create-release

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development