Flume
  1. Flume
  2. FLUME-1262

Move doc generation to a different profile

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.4.0
    • Component/s: Docs
    • Labels:
      None

      Description

      Moving to Sphinx has caused our build memory requirements to increase, causing build failures if MAVEN_OPTS does not set higher PermSpace. We should move it to a separate profile, to remove this requirement for normal dev builds and also speed up the build(this is not a major concern - doc generation itself does not take much time).

      1. FLUME-1262-3.patch
        7 kB
        Mike Percy
      2. FLUME-1262-2.patch
        7 kB
        Mike Percy
      3. FLUME-1262-1.patch
        5 kB
        Mike Percy
      4. FLUME-1262-0.patch
        6 kB
        Mike Percy

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          As Ralph suggested on the dev thread, we can do it when mvn release/site is done - thus not requiring a different profile.

          Show
          Hari Shreedharan added a comment - As Ralph suggested on the dev thread, we can do it when mvn release/site is done - thus not requiring a different profile.
          Hide
          Ralph Goers added a comment -

          After determining that Sphinx or the maven-sphinx-plugin is the culprit for causing the OutOfMemory errors during the build I would recommend two things: a) either revert FLUME-1242 which introduced Sphinx or switch to another document generation technology and b) switch to a "normal" maven site build where the site is only built when you run "mvn site" and deployed with mvn site:deploy or mvn site:stage-deploy.

          Show
          Ralph Goers added a comment - After determining that Sphinx or the maven-sphinx-plugin is the culprit for causing the OutOfMemory errors during the build I would recommend two things: a) either revert FLUME-1242 which introduced Sphinx or switch to another document generation technology and b) switch to a "normal" maven site build where the site is only built when you run "mvn site" and deployed with mvn site:deploy or mvn site:stage-deploy.
          Hide
          Ralph Goers added a comment -

          I have created branch flume-1262 and have a working site build that fixes this problem, FLUME-1256 and FLUME-1282. It isn't complete yet as most of the sub-projects need and index file created to describe them. Originally I was going to keep the Sphinx part of the build, but after seeing that it didn't integrate well into the project sites I chose to incorporate the documentation into the normal site build.

          To build the project just run "mvn package" or "mvn install" as normal. To build the web site run "mvn site" and then "mvn site:deploy -DsiteUrlDeployment=file:///Users/xxxx/flume" where xxxx is your userid (on a unix system anyway). To view the site browse file ~/flume/index.html.

          The build should also support using "mvn release:prepare" and "mvn release:perform" to do a release.

          Show
          Ralph Goers added a comment - I have created branch flume-1262 and have a working site build that fixes this problem, FLUME-1256 and FLUME-1282 . It isn't complete yet as most of the sub-projects need and index file created to describe them. Originally I was going to keep the Sphinx part of the build, but after seeing that it didn't integrate well into the project sites I chose to incorporate the documentation into the normal site build. To build the project just run "mvn package" or "mvn install" as normal. To build the web site run "mvn site" and then "mvn site:deploy -DsiteUrlDeployment= file:///Users/xxxx/flume " where xxxx is your userid (on a unix system anyway). To view the site browse file ~/flume/index.html. The build should also support using "mvn release:prepare" and "mvn release:perform" to do a release.
          Hide
          Mike Percy added a comment -

          Patch to enable skipping generation of all docs via -DskipDocs mvn flag

          Show
          Mike Percy added a comment - Patch to enable skipping generation of all docs via -DskipDocs mvn flag
          Hide
          Mike Percy added a comment -

          bump, needs review

          Show
          Mike Percy added a comment - bump, needs review
          Hide
          Hari Shreedharan added a comment -

          Can we also move the site/javadoc generation a different profile, so it can be executed after the build? This will fix the build failure in a new version issue (as you can see happened in case of 1.3.0 builds etc). ?

          Show
          Hari Shreedharan added a comment - Can we also move the site/javadoc generation a different profile, so it can be executed after the build? This will fix the build failure in a new version issue (as you can see happened in case of 1.3.0 builds etc). ?
          Hide
          Hari Shreedharan added a comment -

          Basically activation be explicit using the profile name, rather than have a -DskipDocs (this can be done by removing the activation tags)?

          Show
          Hari Shreedharan added a comment - Basically activation be explicit using the profile name, rather than have a -DskipDocs (this can be done by removing the activation tags)?
          Hide
          Mike Percy added a comment -

          Updated patch

          Show
          Mike Percy added a comment - Updated patch
          Hide
          Mike Percy added a comment -

          Hari, thanks for the feedback. I will update the release docs with -Psite once this is committed.

          Show
          Mike Percy added a comment - Hari, thanks for the feedback. I will update the release docs with -Psite once this is committed.
          Hide
          Hari Shreedharan added a comment -

          Mike Percy Generally looks good. Can we also move the site generation to the site profile? The patch seems to move everything except that.

          Show
          Hari Shreedharan added a comment - Mike Percy Generally looks good. Can we also move the site generation to the site profile? The patch seems to move everything except that.
          Hide
          Edward Sargisson added a comment -

          This is actually a bug and not just an improvement.

          Without this patch, the Maven build will attempt to create Javadoc based on the version of the artifacts it finds in the repository - and then goes on to build new versions. This means that the Javadoc is always out of date unless you build the same code twice.

          It also makes the code impossible for new developers to build without figuring out how to disable the javadoc generation.

          Show
          Edward Sargisson added a comment - This is actually a bug and not just an improvement. Without this patch, the Maven build will attempt to create Javadoc based on the version of the artifacts it finds in the repository - and then goes on to build new versions. This means that the Javadoc is always out of date unless you build the same code twice. It also makes the code impossible for new developers to build without figuring out how to disable the javadoc generation.
          Hide
          Israel Ekpo added a comment -

          I updated it to Improvement based on the description of the issue.

          Do you have the steps for disabling it?

          It would be nice for us to document that.

          Show
          Israel Ekpo added a comment - I updated it to Improvement based on the description of the issue. Do you have the steps for disabling it? It would be nice for us to document that.
          Hide
          Hari Shreedharan added a comment -

          I agree with Edward. This is a bug. The issue is that javadocs are generated from jars before the jars are built, so the javadocs built are outdated. Also, when a version bump is done, then build goes crazy, because there are no jars of that version in the .m2

          Show
          Hari Shreedharan added a comment - I agree with Edward. This is a bug. The issue is that javadocs are generated from jars before the jars are built, so the javadocs built are outdated. Also, when a version bump is done, then build goes crazy, because there are no jars of that version in the .m2
          Hide
          Israel Ekpo added a comment -

          Updating type to bug based on discussion.

          Show
          Israel Ekpo added a comment - Updating type to bug based on discussion.
          Hide
          Roshan Naik added a comment -

          This patch does not seem to apply anymore.

          Show
          Roshan Naik added a comment - This patch does not seem to apply anymore.
          Hide
          Roshan Naik added a comment -

          Would this patch essentially imply the following two steps in the release build ?

          • mvn clean install -DskipTests
          • mvn install -P site
          Show
          Roshan Naik added a comment - Would this patch essentially imply the following two steps in the release build ? mvn clean install -DskipTests mvn install -P site
          Hide
          Hari Shreedharan added a comment -

          You can do mvn clean install -Psite at once.

          Show
          Hari Shreedharan added a comment - You can do mvn clean install -Psite at once.
          Hide
          Mike Percy added a comment -

          OK I cleaned some junk out of doc generation Maven stuff, so now javadoc, the site plugin, and the sphinx docs can be run separately (via mvn site and mvn javadoc:aggregate-jar) but are not enabled at package time unless you also specify the -Psite profile. I think this should be good to go.

          Show
          Mike Percy added a comment - OK I cleaned some junk out of doc generation Maven stuff, so now javadoc, the site plugin, and the sphinx docs can be run separately (via mvn site and mvn javadoc:aggregate-jar) but are not enabled at package time unless you also specify the -Psite profile. I think this should be good to go.
          Hide
          Hari Shreedharan added a comment -

          +1. Looks good Mike. Could you also update the "How to Release" wiki page.

          Show
          Hari Shreedharan added a comment - +1. Looks good Mike. Could you also update the "How to Release" wiki page.
          Hide
          Hari Shreedharan added a comment -

          Patch committed, rev: 66e707f.

          Please update the wiki with the command to build the release packages, and also the website - you will need to update the section that describes moving the API docs as well.
          Thanks Mike!

          Show
          Hari Shreedharan added a comment - Patch committed, rev: 66e707f. Please update the wiki with the command to build the release packages, and also the website - you will need to update the section that describes moving the API docs as well. Thanks Mike!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #393 (See https://builds.apache.org/job/flume-trunk/393/)
          FLUME-1262. Move doc generation to a different profile (Revision 66e707fe8e38ac8a28058b845bc28164442d55f6)

          Result = UNSTABLE
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=66e707fe8e38ac8a28058b845bc28164442d55f6
          Files :

          • flume-ng-node/pom.xml
          • pom.xml
          • flume-ng-dist/src/main/assembly/bin.xml
          Show
          Hudson added a comment - Integrated in flume-trunk #393 (See https://builds.apache.org/job/flume-trunk/393/ ) FLUME-1262 . Move doc generation to a different profile (Revision 66e707fe8e38ac8a28058b845bc28164442d55f6) Result = UNSTABLE hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=66e707fe8e38ac8a28058b845bc28164442d55f6 Files : flume-ng-node/pom.xml pom.xml flume-ng-dist/src/main/assembly/bin.xml
          Hide
          Mike Percy added a comment -

          Just to follow up, I updated the docs @ https://cwiki.apache.org/confluence/display/FLUME/How+to+Release ... I didn't have to mention anything about moving the RST docs to the web site since I think that part stays the same.

          Show
          Mike Percy added a comment - Just to follow up, I updated the docs @ https://cwiki.apache.org/confluence/display/FLUME/How+to+Release ... I didn't have to mention anything about moving the RST docs to the web site since I think that part stays the same.

            People

            • Assignee:
              Mike Percy
              Reporter:
              Hari Shreedharan
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development