Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7281

Fix various issues in (Maven) release infrastructure

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.1, 1.4.0
    • Fix Version/s: 1.4.0, 1.3.2
    • Component/s: Build System
    • Labels:
      None

      Description

      I discovered couple of issues while getting ready for release 1.3.2:

      • some old, misleading release scripts and release README
      • the maven-release-plugin is not correctly configured for doing actual releases with it
      • the quickstarts are not configured to depend on the project version and thus require manual updating, also of slf4j and log4j versions
      • the maven-javadoc-plugin configuration does not work when using the the maven-release-plugin, that is we have to move the config to the plugin section and out of the release profile

        Issue Links

          Activity

          Hide
          aljoscha Aljoscha Krettek added a comment -

          Implemented on master in
          50a818b1bb74e9442478157bb10c0d5de05ad665
          f1c92f2367183d7e215466b3ff227fff11efc470

          Implemented on release-1.3 in
          62b67876cec0ef13e35eec242a01029899b916b4
          b5c9617b41eb38c104ea5c20f15a1f937c591b40

          Show
          aljoscha Aljoscha Krettek added a comment - Implemented on master in 50a818b1bb74e9442478157bb10c0d5de05ad665 f1c92f2367183d7e215466b3ff227fff11efc470 Implemented on release-1.3 in 62b67876cec0ef13e35eec242a01029899b916b4 b5c9617b41eb38c104ea5c20f15a1f937c591b40
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/4407

          @zentol Yes, I'm planning to remove the old script and the documentation for the new scripts is taking shape here: https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release

          The scripts are not yet described there but I will fill this in as I go. I think it's already quite the improvement compared to the old release "guide": https://cwiki.apache.org/confluence/display/FLINK/Releasing

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4407 @zentol Yes, I'm planning to remove the old script and the documentation for the new scripts is taking shape here: https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release The scripts are not yet described there but I will fill this in as I go. I think it's already quite the improvement compared to the old release "guide": https://cwiki.apache.org/confluence/display/FLINK/Releasing
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha closed the pull request at:

          https://github.com/apache/flink/pull/4407

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/4407
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/4407

          Merged

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4407 Merged
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on the issue:

          https://github.com/apache/flink/pull/4407

          the changes for FLINK-7281 look good.

          As for FLINK-7290, you didn't adjust the existing release script. Do you intend to delete it or adjust it to call the module scripts? (There is after all no documentation on how to use the scripts, like their order)

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on the issue: https://github.com/apache/flink/pull/4407 the changes for FLINK-7281 look good. As for FLINK-7290 , you didn't adjust the existing release script. Do you intend to delete it or adjust it to call the module scripts? (There is after all no documentation on how to use the scripts, like their order)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/4407

          @zentol I will remove some changes from this so that only the Quickstart changes remain and the removal of the old release script and README. I think using `maven-release-plugin` is currently not a good option so I will leave that as is. What do you think?

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4407 @zentol I will remove some changes from this so that only the Quickstart changes remain and the removal of the old release script and README. I think using `maven-release-plugin` is currently not a good option so I will leave that as is. What do you think?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129824759

          — Diff: tools/force-shading/pom.xml —
          @@ -93,11 +93,13 @@ under the License.
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-release-plugin</artifactId>

          • <version>2.1</version>
            + <version>2.5.3</version>
            <configuration>
          • <mavenExecutorId>forked-path</mavenExecutorId>
          • <useReleaseProfile>false</useReleaseProfile>
          • <arguments>$ {arguments}

            -Psonatype-oss-release</arguments>
            + <releaseProfiles>release</releaseProfiles>
            + <preparationGoals>clean install</preparationGoals>
            + <goals>deploy</goals>
            + <autoVersionSubmodules>true</autoVersionSubmodules>
            + <pushChanges>false</pushChanges>

              • End diff –

          `autoVersionSubmodules` tells it to update the version in all the submodules. `pushChanges` instructs it to not early push new branches, commits, tags to git. Otherwise it does that, which seems a bit crazy.

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129824759 — Diff: tools/force-shading/pom.xml — @@ -93,11 +93,13 @@ under the License. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-release-plugin</artifactId> <version>2.1</version> + <version>2.5.3</version> <configuration> <mavenExecutorId>forked-path</mavenExecutorId> <useReleaseProfile>false</useReleaseProfile> <arguments>$ {arguments} -Psonatype-oss-release</arguments> + <releaseProfiles>release</releaseProfiles> + <preparationGoals>clean install</preparationGoals> + <goals>deploy</goals> + <autoVersionSubmodules>true</autoVersionSubmodules> + <pushChanges>false</pushChanges> End diff – `autoVersionSubmodules` tells it to update the version in all the submodules. `pushChanges` instructs it to not early push new branches, commits, tags to git. Otherwise it does that, which seems a bit crazy.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129824585

          — Diff: pom.xml —
          @@ -1246,14 +1247,32 @@ under the License.
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-release-plugin</artifactId>

          • <version>2.1</version>
            + <version>2.5.3</version>
            <configuration>
          • <mavenExecutorId>forked-path</mavenExecutorId>
          • <useReleaseProfile>false</useReleaseProfile>
          • <arguments>$ {arguments}

            -Psonatype-oss-release</arguments>
            + <releaseProfiles>release</releaseProfiles>
            + <preparationGoals>clean install</preparationGoals>
            + <goals>deploy</goals>
            + <autoVersionSubmodules>true</autoVersionSubmodules>
            + <pushChanges>false</pushChanges>
            </configuration>
            </plugin>

          + <!-- Rename the artifact produced by the Apache parent's
          + source release. -->
          + <plugin>
          + <groupId>org.apache.maven.plugins</groupId>
          + <artifactId>maven-assembly-plugin</artifactId>
          — End diff –

          Not, sure, actually. I'm setting it manually to make it resemble our current file naming.

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129824585 — Diff: pom.xml — @@ -1246,14 +1247,32 @@ under the License. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-release-plugin</artifactId> <version>2.1</version> + <version>2.5.3</version> <configuration> <mavenExecutorId>forked-path</mavenExecutorId> <useReleaseProfile>false</useReleaseProfile> <arguments>$ {arguments} -Psonatype-oss-release</arguments> + <releaseProfiles>release</releaseProfiles> + <preparationGoals>clean install</preparationGoals> + <goals>deploy</goals> + <autoVersionSubmodules>true</autoVersionSubmodules> + <pushChanges>false</pushChanges> </configuration> </plugin> + <!-- Rename the artifact produced by the Apache parent's + source release. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-assembly-plugin</artifactId> — End diff – Not, sure, actually. I'm setting it manually to make it resemble our current file naming.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129824394

          — Diff: pom.xml —
          @@ -802,20 +802,6 @@ under the License.
          </executions>
          </plugin>
          </plugins>

          • <pluginManagement>
              • End diff –

          As mentioned above, it's not actually needed for our current release process but I'd like to make it usable for that. Apparently how you use it with the Apache parent pom is to put it into the `pluginManagement` section.

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129824394 — Diff: pom.xml — @@ -802,20 +802,6 @@ under the License. </executions> </plugin> </plugins> <pluginManagement> End diff – As mentioned above, it's not actually needed for our current release process but I'd like to make it usable for that. Apparently how you use it with the Apache parent pom is to put it into the `pluginManagement` section.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129824175

          — Diff: pom.xml —
          @@ -1257,6 +1243,17 @@ under the License.
          <pluginManagement>
          <plugins>

          + <plugin>
          + <groupId>org.apache.maven.plugins</groupId>
          + <artifactId>maven-release-plugin</artifactId>
          — End diff –

          We were actually not using this before, it was just a leftover in there. It's not even required for our release process, i.e. in the release profile.

          However, the plugin can be used to do the stuff that we do in scripts and manually. For example , I can do
          ```
          mvn release:branch \
          -DbranchName=$

          {BRANCH_NAME}

          \
          -DdevelopmentVersion=$

          {DEVELOPMENT_VERSION}

          ```
          to update the version on master and fork of the branch for a newly released version.

          Then I could use
          ```
          mvn release:prepare \
          -Dresume=false \
          -DreleaseVersion=$

          {VERSION}

          \
          -Dtag=$

          {TAG}

          \
          -DupdateWorkingCopyVersions=false
          ```
          and
          ```
          mvn release:perform
          ```
          to build the actual release, create the source release, upload the (signed) jars to sonatype and create the release tag.

          (I was experimenting with this when preparing for the next release and our release infrastructure is very interesting.)

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129824175 — Diff: pom.xml — @@ -1257,6 +1243,17 @@ under the License. <pluginManagement> <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-release-plugin</artifactId> — End diff – We were actually not using this before, it was just a leftover in there. It's not even required for our release process, i.e. in the release profile. However, the plugin can be used to do the stuff that we do in scripts and manually. For example , I can do ``` mvn release:branch \ -DbranchName=$ {BRANCH_NAME} \ -DdevelopmentVersion=$ {DEVELOPMENT_VERSION} ``` to update the version on master and fork of the branch for a newly released version. Then I could use ``` mvn release:prepare \ -Dresume=false \ -DreleaseVersion=$ {VERSION} \ -Dtag=$ {TAG} \ -DupdateWorkingCopyVersions=false ``` and ``` mvn release:perform ``` to build the actual release, create the source release, upload the (signed) jars to sonatype and create the release tag. (I was experimenting with this when preparing for the next release and our release infrastructure is very interesting .)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129808895

          — Diff: tools/force-shading/pom.xml —
          @@ -93,11 +93,13 @@ under the License.
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-release-plugin</artifactId>

          • <version>2.1</version>
            + <version>2.5.3</version>
            <configuration>
          • <mavenExecutorId>forked-path</mavenExecutorId>
          • <useReleaseProfile>false</useReleaseProfile>
          • <arguments>$ {arguments}

            -Psonatype-oss-release</arguments>
            + <releaseProfiles>release</releaseProfiles>
            + <preparationGoals>clean install</preparationGoals>
            + <goals>deploy</goals>
            + <autoVersionSubmodules>true</autoVersionSubmodules>
            + <pushChanges>false</pushChanges>

              • End diff –

          could you explain what the last 2 options do?

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129808895 — Diff: tools/force-shading/pom.xml — @@ -93,11 +93,13 @@ under the License. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-release-plugin</artifactId> <version>2.1</version> + <version>2.5.3</version> <configuration> <mavenExecutorId>forked-path</mavenExecutorId> <useReleaseProfile>false</useReleaseProfile> <arguments>$ {arguments} -Psonatype-oss-release</arguments> + <releaseProfiles>release</releaseProfiles> + <preparationGoals>clean install</preparationGoals> + <goals>deploy</goals> + <autoVersionSubmodules>true</autoVersionSubmodules> + <pushChanges>false</pushChanges> End diff – could you explain what the last 2 options do?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129808825

          — Diff: pom.xml —
          @@ -1246,14 +1247,32 @@ under the License.
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-release-plugin</artifactId>

          • <version>2.1</version>
            + <version>2.5.3</version>
            <configuration>
          • <mavenExecutorId>forked-path</mavenExecutorId>
          • <useReleaseProfile>false</useReleaseProfile>
          • <arguments>$ {arguments}

            -Psonatype-oss-release</arguments>
            + <releaseProfiles>release</releaseProfiles>
            + <preparationGoals>clean install</preparationGoals>
            + <goals>deploy</goals>
            + <autoVersionSubmodules>true</autoVersionSubmodules>
            + <pushChanges>false</pushChanges>
            </configuration>
            </plugin>

          + <!-- Rename the artifact produced by the Apache parent's
          + source release. -->
          + <plugin>
          + <groupId>org.apache.maven.plugins</groupId>
          + <artifactId>maven-assembly-plugin</artifactId>
          — End diff –

          what would the name be without this?

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129808825 — Diff: pom.xml — @@ -1246,14 +1247,32 @@ under the License. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-release-plugin</artifactId> <version>2.1</version> + <version>2.5.3</version> <configuration> <mavenExecutorId>forked-path</mavenExecutorId> <useReleaseProfile>false</useReleaseProfile> <arguments>$ {arguments} -Psonatype-oss-release</arguments> + <releaseProfiles>release</releaseProfiles> + <preparationGoals>clean install</preparationGoals> + <goals>deploy</goals> + <autoVersionSubmodules>true</autoVersionSubmodules> + <pushChanges>false</pushChanges> </configuration> </plugin> + <!-- Rename the artifact produced by the Apache parent's + source release. --> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-assembly-plugin</artifactId> — End diff – what would the name be without this?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129808661

          — Diff: pom.xml —
          @@ -802,20 +802,6 @@ under the License.
          </executions>
          </plugin>
          </plugins>

          • <pluginManagement>
              • End diff –

          what's the benefit of moving it out of the release profile? The profile still exists, and the plugin should only be used when making a release, no?

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129808661 — Diff: pom.xml — @@ -802,20 +802,6 @@ under the License. </executions> </plugin> </plugins> <pluginManagement> End diff – what's the benefit of moving it out of the release profile? The profile still exists, and the plugin should only be used when making a release, no?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4407#discussion_r129807756

          — Diff: pom.xml —
          @@ -1257,6 +1243,17 @@ under the License.
          <pluginManagement>
          <plugins>

          + <plugin>
          + <groupId>org.apache.maven.plugins</groupId>
          + <artifactId>maven-release-plugin</artifactId>
          — End diff –

          given that it is only defined in the plugin management section i was wondering whether we need this plugin at all.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/4407#discussion_r129807756 — Diff: pom.xml — @@ -1257,6 +1243,17 @@ under the License. <pluginManagement> <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-release-plugin</artifactId> — End diff – given that it is only defined in the plugin management section i was wondering whether we need this plugin at all.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user aljoscha opened a pull request:

          https://github.com/apache/flink/pull/4407

          FLINK-7281 Fix various issues in (Maven) release infrastructure

            1. What is the purpose of the change

          Fix various issues in the release infrastructure and Maven.

            1. Brief change log
          • remove old, misleading release scripts and release README
          • configure the maven-release-plugin for doing actual releases with it
          • make the quickstarts depend on project versions
          • configure maven-javadoc-plugin to work with maven-release-plugin (doc linting was not disabled)
            1. Verifying this change

          This change is a trivial rework / code cleanup without any test coverage.

            1. Does this pull request potentially affect one of the following parts:

          no

            1. Documentation
          • Does this pull request introduce a new feature? (no)

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/aljoscha/flink release-1.3

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/4407.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #4407


          commit b37c29feb3814969031b573bc73dcd09c807744f
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-07-25T13:07:59Z

          FLINK-7281 Remove old and misleading release scripts/README

          commit 6d6b5289de72016ea65a4f869cda3ef58a26a5d4
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-07-25T13:30:26Z

          FLINK-7281 Make quickstart versions depend on project versions

          commit ded58c9bf0a21e5205d514e010b515b1ef1e7f2f
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-07-25T13:30:42Z

          FLINK-7281 Bump maven-release-plugin version and make it work

          commit 1a5cecab8f91cadbdd86c6194ef6276f1cc5df83
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-07-26T10:10:15Z

          FLINK-7281 Move maven-javadoc-config out of execution blocks

          commit 085852028837cf9d7835a6120385ea665e8510e3
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-07-26T12:19:49Z

          FLINK-7281 Move maven-release-plugin out of release profile


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/4407 FLINK-7281 Fix various issues in (Maven) release infrastructure What is the purpose of the change Fix various issues in the release infrastructure and Maven. Brief change log remove old, misleading release scripts and release README configure the maven-release-plugin for doing actual releases with it make the quickstarts depend on project versions configure maven-javadoc-plugin to work with maven-release-plugin (doc linting was not disabled) Verifying this change This change is a trivial rework / code cleanup without any test coverage. Does this pull request potentially affect one of the following parts: no Documentation Does this pull request introduce a new feature? (no) You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink release-1.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4407.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4407 commit b37c29feb3814969031b573bc73dcd09c807744f Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-07-25T13:07:59Z FLINK-7281 Remove old and misleading release scripts/README commit 6d6b5289de72016ea65a4f869cda3ef58a26a5d4 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-07-25T13:30:26Z FLINK-7281 Make quickstart versions depend on project versions commit ded58c9bf0a21e5205d514e010b515b1ef1e7f2f Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-07-25T13:30:42Z FLINK-7281 Bump maven-release-plugin version and make it work commit 1a5cecab8f91cadbdd86c6194ef6276f1cc5df83 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-07-26T10:10:15Z FLINK-7281 Move maven-javadoc-config out of execution blocks commit 085852028837cf9d7835a6120385ea665e8510e3 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-07-26T12:19:49Z FLINK-7281 Move maven-release-plugin out of release profile

            People

            • Assignee:
              aljoscha Aljoscha Krettek
              Reporter:
              aljoscha Aljoscha Krettek
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development