Hadoop Common
  1. Hadoop Common
  2. HADOOP-7069

Replace forrest with supported framework

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: documentation
    • Labels:
      None

      Description

      It's time to burn down the forrest. Apache forrest, which is used to generate the documentation for all three subprojects, has not had a release in several years (0.8, the version we use was released April 18, 2007), and requires JDK5, which was EOL'ed in November 2009. Since it doesn't seem likely Forrest will be developed any more, and JDK5 is not shipped with recent OSX versions, or included by default in most linux distros, we should look to find a new documentation system and convert the current docs to it.

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Eli Collins made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 2.0.0 [ 12320352 ]
          Fix Version/s 0.24.0 [ 12317652 ]
          Resolution Fixed [ 1 ]
          Konstantin Boudnik made changes -
          Link This issue is cloned as HADOOP-8399 [ HADOOP-8399 ]
          Konstantin Boudnik made changes -
          Link This issue is cloned as HADOOP-8399 [ HADOOP-8399 ]
          Arun C Murthy made changes -
          Fix Version/s 0.24.0 [ 12317652 ]
          Fix Version/s 0.23.0 [ 12315569 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to BIGTOP-65 [ BIGTOP-65 ]
          Hide
          Scott Carey added a comment -

          The mavenization effort is a huge amount of work that would be great to get done, but it's probably too big (x3 - thanks project split!)

          Its nothing like 3x the work, if done right. Most of the work would end up in a parent project pom.xml(s) that all 3 projects would inherit from. They could then be built together or independently.

          For Avro, the dropping of the JDK5 requirement is huge and we may not switch soon without that as a motivating force. But other tools, like Sphynx (http://sphinx.pocoo.org/) are interesting in part because they can have a custom look/feel easily and ... look good We also have to support build artifacts and documentation for many languages so the python origins of something like Sphynx isn't a barrier to entry.

          Maven provides a lot out of the box, but it is ugly and the site will look like a maven project unless customized. Its great for what it does by default trivially but is built for convention and not customization. Build a project with Maven and you get a site for free with zero effort and some extras with little work at all. But it is focused on build artifacts and a few faq-like pages. Build a maven plugin and the docs it generates are beautiful right out of the box – coming from annotations /javadoc in your classes. Want to add a bunch of custom documentation or document types? significantly change look/feel? – not so trivial.

          Show
          Scott Carey added a comment - The mavenization effort is a huge amount of work that would be great to get done, but it's probably too big (x3 - thanks project split!) Its nothing like 3x the work, if done right. Most of the work would end up in a parent project pom.xml(s) that all 3 projects would inherit from. They could then be built together or independently. For Avro, the dropping of the JDK5 requirement is huge and we may not switch soon without that as a motivating force. But other tools, like Sphynx ( http://sphinx.pocoo.org/ ) are interesting in part because they can have a custom look/feel easily and ... look good We also have to support build artifacts and documentation for many languages so the python origins of something like Sphynx isn't a barrier to entry. Maven provides a lot out of the box, but it is ugly and the site will look like a maven project unless customized. Its great for what it does by default trivially but is built for convention and not customization. Build a project with Maven and you get a site for free with zero effort and some extras with little work at all. But it is focused on build artifacts and a few faq-like pages. Build a maven plugin and the docs it generates are beautiful right out of the box – coming from annotations /javadoc in your classes. Want to add a bunch of custom documentation or document types? significantly change look/feel? – not so trivial.
          Hide
          Konstantin Boudnik added a comment -

          As has been discussed separately HADOOP-7075 is a dup of HADOOP-7072. The latter has been committed to the trunk and 0.21, 0.22 branches. I want to thank Carl for pointing to the appropriate PIG issue which provided the idea for the fix.

          Show
          Konstantin Boudnik added a comment - As has been discussed separately HADOOP-7075 is a dup of HADOOP-7072 . The latter has been committed to the trunk and 0.21, 0.22 branches. I want to thank Carl for pointing to the appropriate PIG issue which provided the idea for the fix.
          Hide
          Carl Steinbach added a comment -

          There's now a patch available in HADOOP-7075 that fixes the Forrest JDK6 problem.

          Forrest is dog slow. The hadoop tar build spends most of it's time building docs.

          With my fix for HADOOP-7075 in place it takes me just under a minute to build the docs
          target. This seems pretty reasonable to me.

          Show
          Carl Steinbach added a comment - There's now a patch available in HADOOP-7075 that fixes the Forrest JDK6 problem. Forrest is dog slow. The hadoop tar build spends most of it's time building docs. With my fix for HADOOP-7075 in place it takes me just under a minute to build the docs target. This seems pretty reasonable to me.
          Carl Steinbach made changes -
          Link This issue is related to HADOOP-7075 [ HADOOP-7075 ]
          Hide
          Konstantin Boudnik added a comment -

          The mavenization effort is a huge amount of work that would be great to get done, but it's probably too big (x3 - thanks project split!

          When I was leaving Giri was finishing his Maven patch for Common. I assume it is done by now, but I don't know what's the status of it.

          To get a sense of what kind of complexity maven site plugin can handle take a look at Apache maven http://maven.apache.org/

          Show
          Konstantin Boudnik added a comment - The mavenization effort is a huge amount of work that would be great to get done, but it's probably too big (x3 - thanks project split! When I was leaving Giri was finishing his Maven patch for Common. I assume it is done by now, but I don't know what's the status of it. To get a sense of what kind of complexity maven site plugin can handle take a look at Apache maven http://maven.apache.org/
          Hide
          Jakob Homan added a comment -

          It'd be great if we can get forrest working with Java 6, sure thing. But given the lack of activity in the Forrest project over the past nearly four years, it's probably prudent to move sooner than later. I played with textile a bit the other night and actually quite liked it. It's worth comparing it (and others) against maven's site generation. I would tend toward textile (or something like it) rather than whatever Maven happens to provide, on the assumption that a stand-alone tool would provide more features and flexibility.

          The mavenization effort is a huge amount of work that would be great to get done, but it's probably too big (x3 - thanks project split!) to be done without a pressing need from one of the commercial interests contributing to Hadoop. It would be great to know if that's being planned. I'm not aware that it is.

          Show
          Jakob Homan added a comment - It'd be great if we can get forrest working with Java 6, sure thing. But given the lack of activity in the Forrest project over the past nearly four years, it's probably prudent to move sooner than later. I played with textile a bit the other night and actually quite liked it. It's worth comparing it (and others) against maven's site generation. I would tend toward textile (or something like it) rather than whatever Maven happens to provide, on the assumption that a stand-alone tool would provide more features and flexibility. The mavenization effort is a huge amount of work that would be great to get done, but it's probably too big (x3 - thanks project split!) to be done without a pressing need from one of the commercial interests contributing to Hadoop. It would be great to know if that's being planned. I'm not aware that it is.
          Hide
          Konstantin Boudnik added a comment -

          There's no question we need to get rid of Forrest. However. considering Carl's post:

          • in the short term we can get rid of annoying JDK5 dependency (makes me happy)
          • in a longer term to switch to Maven site generation without going having to convert to textile format and facing an unknown effort of support it under maven'ized build.
          Show
          Konstantin Boudnik added a comment - There's no question we need to get rid of Forrest. However. considering Carl's post: in the short term we can get rid of annoying JDK5 dependency (makes me happy) in a longer term to switch to Maven site generation without going having to convert to textile format and facing an unknown effort of support it under maven'ized build.
          Hide
          Eli Collins added a comment -

          Forrest is dog slow. The hadoop tar build spends most of it's time building docs.

          Show
          Eli Collins added a comment - Forrest is dog slow. The hadoop tar build spends most of it's time building docs.
          Hide
          Carl Steinbach added a comment -

          Sorry, what I meant is that it's possible to make Forrest work with JDK6, but I think you get the idea.

          Show
          Carl Steinbach added a comment - Sorry, what I meant is that it's possible to make Forrest work with JDK6, but I think you get the idea.
          Hide
          Carl Steinbach added a comment -

          You can make Forrest work with JDK5 by making a one line change to your forrest.properties file.
          See FOR-984 or PIG-1508.

          Are there any problems with Forrest besides the JDK5 issue?

          Show
          Carl Steinbach added a comment - You can make Forrest work with JDK5 by making a one line change to your forrest.properties file. See FOR-984 or PIG-1508 . Are there any problems with Forrest besides the JDK5 issue?
          Hide
          Owen O'Malley added a comment -

          I've used textile via Jekyll and found it very nice. You can see the results over at http://yahoo.github.com/hadoop-common/. I found the layout very easy to use and debug.

          Show
          Owen O'Malley added a comment - I've used textile via Jekyll and found it very nice. You can see the results over at http://yahoo.github.com/hadoop-common/ . I found the layout very easy to use and debug.
          Jeff Hammerbacher made changes -
          Link This issue is related to HADOOP-6671 [ HADOOP-6671 ]
          Hide
          Jakob Homan added a comment -

          The last comment on HADOOP-6671, was back in April, 8 months ago. I don't know of any reason to expect there to be any movement on it now.

          Show
          Jakob Homan added a comment - The last comment on HADOOP-6671 , was back in April, 8 months ago. I don't know of any reason to expect there to be any movement on it now.
          Hide
          Konstantin Boudnik added a comment -

          I believe the mavenization effort is happening or isn't it? If it does I'd vote to bear the forrest pain for a while longer and then dump it all together without intermittent steps.

          Show
          Konstantin Boudnik added a comment - I believe the mavenization effort is happening or isn't it? If it does I'd vote to bear the forrest pain for a while longer and then dump it all together without intermittent steps.
          Hide
          Jeff Hammerbacher added a comment -
          Show
          Jeff Hammerbacher added a comment - A few other projects have tried this one too: Avro: https://issues.apache.org/jira/browse/AVRO-319 Hive: https://issues.apache.org/jira/browse/HIVE-1135
          Jeff Hammerbacher made changes -
          Link This issue is related to AVRO-319 [ AVRO-319 ]
          Jakob Homan made changes -
          Link This issue is related to ZOOKEEPER-925 [ ZOOKEEPER-925 ]
          Jakob Homan made changes -
          Field Original Value New Value
          Link This issue is related to WHIRR-19 [ WHIRR-19 ]
          Hide
          Jakob Homan added a comment -

          Both Whirr and ZooKeeper have recently looked at this issue and come to different conclusions:

          Perhaps we should take the benefit of Ben and Pat's toil and evaluate textile first, although it seems to have its share of problems as well.

          Show
          Jakob Homan added a comment - Both Whirr and ZooKeeper have recently looked at this issue and come to different conclusions: Whirr ( WHIRR-19 ) went with maven site generation, but this seems out of reach until the build is mavenized, which isn't likely to happen soon. ZooKeeper ( ZOOKEEPER-925 ) went through the pain of trying to convert its existing forrest docs and eventually chose textile ( https://issues.apache.org/jira/browse/ZOOKEEPER-925?focusedCommentId=12971846&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12971846 ) Perhaps we should take the benefit of Ben and Pat's toil and evaluate textile first, although it seems to have its share of problems as well.
          Jakob Homan created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Jakob Homan
            • Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development