Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2199

Flume builds with new version require mvn install before site can be generated

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: v1.4.0
    • Fix Version/s: None
    • Component/s: Build
    • Labels:
      None

      Description

      At this point, if you change the version for Flume, you need to run a mvn install before you can run with -Psite (or, for that matter, javadoc:javadoc) enabled. This is because the top-level POM in flume.git/pom.xml is both the parent POM and the root of the reactor - since it's the parent, it's got to run before any of the children that inherit from it, but site generation should be running after all the children, so that it probably pulls in the reactor's build of each child module, rather than having to pull in one already installed/deployed before the build starts.

      There are a bunch of other reasons to split parent POM and top-level POM, but that's the biggest one right there.

      Also, the javadoc jar generation is a bit messed up - every module's javadoc jar contains not only its own javadocs but the javadocs for every Flume module it depends on. That, again, may make sense in a site context for the top-level, but not for the individual modules. This results in unnecessary bloat in the javadoc jars, and unnecessary time spent downloading the "*-javadoc-resources.jar" for every dependency each module has, due to how the javadoc plugin works. Also the whole site generation per-module thing, which I am not a fan of in most cases. I don't think it's needed here. Tweaking the site plugin not to run anywhere but the top-level and the javadoc plugin to not do the dependency aggregation anywhere but the top-level should make a big difference on build speed.

      1. FLUME-2199.patch
        31 kB
        Andrew Bayer

        Issue Links

          Activity

          Hide
          abayer Andrew Bayer added a comment -

          This patch creates a separate flume-parent/pom.xml, renames the top-level POM to "flume-project", gets rid of the dependencyManagement section for the flume artifacts from the parent in favor of specifying the version as project.version whenever they're used (which makes a bunch of things smoother), sets up proper relative paths for parent POMs everywhere, only runs the site plugin for the top-level POM, and only does the dependency aggregation for javadoc in the top-level POM.

          Net result is a drop in "mvn clean source:jar javadoc:javadoc install -DskipTests -Psite" from 7m52s to 2m48s.

          Show
          abayer Andrew Bayer added a comment - This patch creates a separate flume-parent/pom.xml, renames the top-level POM to "flume-project", gets rid of the dependencyManagement section for the flume artifacts from the parent in favor of specifying the version as project.version whenever they're used (which makes a bunch of things smoother), sets up proper relative paths for parent POMs everywhere, only runs the site plugin for the top-level POM, and only does the dependency aggregation for javadoc in the top-level POM. Net result is a drop in "mvn clean source:jar javadoc:javadoc install -DskipTests -Psite" from 7m52s to 2m48s.
          Hide
          aprabhakar Arvind Prabhakar added a comment -

          Thanks for the patch Andrew. Do you mind publishing a review request?

          Show
          aprabhakar Arvind Prabhakar added a comment - Thanks for the patch Andrew. Do you mind publishing a review request?
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Andrew Bayer Thanks for the patch. Looks like the patch does not apply for flume-ng-dist directory. Also this patch seems to be not on trunk. This patch assumes the presence of a flume-parent directory:

          diff --git flume-parent/pom.xml flume-parent/pom.xml
          index b3c6f4b..c83ae9d 100644
          --- flume-parent/pom.xml
          +++ flume-parent/pom.xml
          @@ -21,7 +21,7 @@ limitations under the License.
             <parent>
               <groupId>org.apache</groupId>
               <artifactId>apache</artifactId>
          -    <version>9</version>
          +    <version>13</version>
             </parent>
          
          Show
          hshreedharan Hari Shreedharan added a comment - Andrew Bayer Thanks for the patch. Looks like the patch does not apply for flume-ng-dist directory. Also this patch seems to be not on trunk. This patch assumes the presence of a flume-parent directory: diff --git flume-parent/pom.xml flume-parent/pom.xml index b3c6f4b..c83ae9d 100644 --- flume-parent/pom.xml +++ flume-parent/pom.xml @@ -21,7 +21,7 @@ limitations under the License. <parent> <groupId>org.apache</groupId> <artifactId>apache</artifactId> - <version>9</version> + <version>13</version> </parent>
          Hide
          brocknoland Brock Noland added a comment -

          Andrew Bayer could you post a full patch? I am working on HIVE-5107 and I'd like to follow this best practice.

          Show
          brocknoland Brock Noland added a comment - Andrew Bayer could you post a full patch? I am working on HIVE-5107 and I'd like to follow this best practice.

            People

            • Assignee:
              abayer Andrew Bayer
              Reporter:
              abayer Andrew Bayer
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development