Solr
  1. Solr
  2. SOLR-555

Autogenerate "user" docs about "plugins" from code

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Our current strategy of using javadocs for "developer" documentation and the wiki for documenting "user" features only gets us so far – among other things, it makes it hard to include the "user" documentation in our releases, but it also results in a disconnect between when code changes and when documentation gets updated.

      in an ideal world, "user" documentation would live in the code right along side the implementation, just like with javadocs – but the standard set of information displayed by javadocs isn't very user friendly. we should find a better way to allow us to "edit" the info about how to use a plugin right along side the code for that plugin and generate user friendly documentation from that.

      1. SOLR-555.patch
        28 kB
        Hoss Man
      2. SOLR-555.patch
        29 kB
        Hoss Man
      3. SOLR-555.patch
        29 kB
        Hoss Man
      4. SOLR-555.patch
        33 kB
        Hoss Man
      5. SOLR-555.patch
        39 kB
        Hoss Man
      6. SOLR-555.patch
        39 kB
        Hoss Man

        Activity

        Hide
        Erick Erickson added a comment -

        Hossman:

        Is there stuff worth keeping here? Doc improvement is really worthwhile, but this has lingered for a looong time. Or should we close it since it hasn't been touched in so long?

        Show
        Erick Erickson added a comment - Hossman: Is there stuff worth keeping here? Doc improvement is really worthwhile, but this has lingered for a looong time. Or should we close it since it hasn't been touched in so long?
        Hide
        Hoss Man added a comment -

        same patch, just updated against trunk.

        I've also updated the generated docs for people to see what the generated doc structure looks like now with the "components" tag and the "smart linking" (if a @see or @link tag refers to a class that is a "plugin" link to it as a plugin, otherwise link to it's normal javadocs)

        Something i didn't really make clear before is that the @solr.component tag isn't specific to the "SearchComponent, it can refer to any class in the tree, and if that class defines some @solr.run.params they will be listed.

        Show
        Hoss Man added a comment - same patch, just updated against trunk. I've also updated the generated docs for people to see what the generated doc structure looks like now with the "components" tag and the "smart linking" (if a @see or @link tag refers to a class that is a "plugin" link to it as a plugin, otherwise link to it's normal javadocs) list of types of plugins: http://people.apache.org/~hossman/tmp/SOLR-555/plugins/ list of instances of a type of plugin: http://people.apache.org/~hossman/tmp/SOLR-555/plugins/org.apache.solr.request.SolrRequestHandler.html A specific instance: http://people.apache.org/~hossman/tmp/SOLR-555/plugins/org.apache.solr.request.SolrRequestHandler.html#org.apache.solr.DemoChildPlugin the regular javadocs for the same class: http://people.apache.org/~hossman/tmp/SOLR-555/org/apache/solr/DemoChildPlugin.html Something i didn't really make clear before is that the @solr.component tag isn't specific to the "SearchComponent, it can refer to any class in the tree, and if that class defines some @solr.run.params they will be listed.
        Hide
        Hoss Man added a comment -

        checkpoint

        includes demo of component info, as well as new code that makes any @link or @see tag which refer to a plugin become a link to that plugin in the plugin docs (but still a link to the javadocs in the javadocs) ... this is kind of a hack right now because it uses static info about which classes are plugins .. just need some refactoring to pass that info along as appropriate.

        Show
        Hoss Man added a comment - checkpoint includes demo of component info, as well as new code that makes any @link or @see tag which refer to a plugin become a link to that plugin in the plugin docs (but still a link to the javadocs in the javadocs) ... this is kind of a hack right now because it uses static info about which classes are plugins .. just need some refactoring to pass that info along as appropriate.
        Hide
        Hoss Man added a comment -

        whoops ... last version was missing a file

        Show
        Hoss Man added a comment - whoops ... last version was missing a file
        Hide
        Hoss Man added a comment -

        added a @solr.component taglet ... any info can be listed in it, and aqt minimum that info will show up in the javadocs (and the plugin docs) but if it contains a

        {@see classname}

        the first sentence summary of that class, and the Runtime Params for that "component" will be listed on the plugin docs as well.

        need to figure out a clean way to make hyperlinking arround the plugin docs work ... wasn't really a big deal before since i was focusing on "flat" plugin docs (plugins directly listed all params inherited from their superclasses) but in the case of components you want just the summary of what the component does for people who are familiar with it; but for people that aren't, you want them to be able to click a link to learn more (without sending them to developer javadocs).

        Show
        Hoss Man added a comment - added a @solr.component taglet ... any info can be listed in it, and aqt minimum that info will show up in the javadocs (and the plugin docs) but if it contains a {@see classname} the first sentence summary of that class, and the Runtime Params for that "component" will be listed on the plugin docs as well. need to figure out a clean way to make hyperlinking arround the plugin docs work ... wasn't really a big deal before since i was focusing on "flat" plugin docs (plugins directly listed all params inherited from their superclasses) but in the case of components you want just the summary of what the component does for people who are familiar with it; but for people that aren't, you want them to be able to click a link to learn more (without sending them to developer javadocs).
        Hide
        Hoss Man added a comment -

        some small improvements on the previous patch...

        • all new files now live in a "plugins/" directory, so they don't pollute the top level javadocs dir.
        • added a {@solr.default}

          for indicating the default value of a solr param

        • added a table of contents to the top of each plugin page with relative links down to the details for each plugin Impl listed.
        Show
        Hoss Man added a comment - some small improvements on the previous patch... all new files now live in a "plugins/" directory, so they don't pollute the top level javadocs dir. added a {@solr.default} for indicating the default value of a solr param added a table of contents to the top of each plugin page with relative links down to the details for each plugin Impl listed.
        Hide
        Hoss Man added a comment -

        Hoss - have you tried XDoclet?

        When i first started experimenting with this idea, i assumed a javadoc doclet would make the most sense .. then i tried making one, and i gave up because it has, by far, the worst "API" I've ever used in my life (caveat: i'm refering solely to Java APIs when i make that statements). So then i started looking at existing third party javadoc doclets, hoping to find a nice one that would already extract all of the useful info into XML so i could use XSLT (there was one, but it's really old and couldn't handle some pretty basic stuff) and then i started looking at other technologies. After experimenting with many of them, I went back and revisited the javadoc doclet approach: came to grips with the cluster f*ck of an API, ignoring the parts of it that were truly abhorent, and using only the core pieces that were really needed (my first attempt was trying to subclass the standard doclet to acutally incorperate the new pages into the existing javadoc navigation) which lead me to the current patch.

        XDoclet, XDoclet2, and QDox (the java parser that XDoclet2 uses) were all on that list of other technologies, i don't remember specifically why i decided not to go with them, but one of the broader issues with most alternate solutions was that since they used good APIs for doing things, they couldn't be hooked into normal javadocs – one of my goals is to try and get to the point where developers write one "doc" for each plugin, it lives in the code and is displayed both in the javadocs for the plugin, as well as in the scaled down "user" docs that this would generated. generating "user" documentation about a plugin using something like XDoclet probably would have been easy – but to get the same information included in the "developer" javadocs would require just as much orthoginal Taglet work as i'm doing now – and simple things like

        {@link}

        tags in the body of a javadoc comment wouldn't work automaticly with XDoclet2 (at least: not as far as i can tell)

        XDoclet2 certianly seems like the best way to auto generate information from java code but for processing javadoc comments, i (unfortunately) couldn't find many solutions better then javadoc doclets and taglets.

        Show
        Hoss Man added a comment - Hoss - have you tried XDoclet? When i first started experimenting with this idea, i assumed a javadoc doclet would make the most sense .. then i tried making one, and i gave up because it has, by far, the worst "API" I've ever used in my life (caveat: i'm refering solely to Java APIs when i make that statements). So then i started looking at existing third party javadoc doclets, hoping to find a nice one that would already extract all of the useful info into XML so i could use XSLT (there was one, but it's really old and couldn't handle some pretty basic stuff) and then i started looking at other technologies. After experimenting with many of them, I went back and revisited the javadoc doclet approach: came to grips with the cluster f*ck of an API, ignoring the parts of it that were truly abhorent, and using only the core pieces that were really needed (my first attempt was trying to subclass the standard doclet to acutally incorperate the new pages into the existing javadoc navigation) which lead me to the current patch. XDoclet, XDoclet2, and QDox (the java parser that XDoclet2 uses) were all on that list of other technologies, i don't remember specifically why i decided not to go with them, but one of the broader issues with most alternate solutions was that since they used good APIs for doing things, they couldn't be hooked into normal javadocs – one of my goals is to try and get to the point where developers write one "doc" for each plugin, it lives in the code and is displayed both in the javadocs for the plugin, as well as in the scaled down "user" docs that this would generated. generating "user" documentation about a plugin using something like XDoclet probably would have been easy – but to get the same information included in the "developer" javadocs would require just as much orthoginal Taglet work as i'm doing now – and simple things like {@link} tags in the body of a javadoc comment wouldn't work automaticly with XDoclet2 (at least: not as far as i can tell) XDoclet2 certianly seems like the best way to auto generate information from java code but for processing javadoc comments, i (unfortunately) couldn't find many solutions better then javadoc doclets and taglets.
        Hide
        Erik Hatcher added a comment -

        Hoss - have you tried XDoclet? I'm not sure where the state of that project is, but I used to do a lot of work with it. I even used it to generate an Ant task reference once upon a time. Been a long long time since I've tinkered with it though, not sure if XDoclet2 is the better starting point or not either.

        XDoclet can make generating stuff from Java code a lot easier than writing your own doclet.

        Show
        Erik Hatcher added a comment - Hoss - have you tried XDoclet? I'm not sure where the state of that project is, but I used to do a lot of work with it. I even used it to generate an Ant task reference once upon a time. Been a long long time since I've tinkered with it though, not sure if XDoclet2 is the better starting point or not either. XDoclet can make generating stuff from Java code a lot easier than writing your own doclet.
        Hide
        Hoss Man added a comment -

        idea from Bram de Jong on solr-user: include info on the default value for all params as well (why didn't I think of that?!?!?!)

        Show
        Hoss Man added a comment - idea from Bram de Jong on solr-user: include info on the default value for all params as well (why didn't I think of that?!?!?!)
        Hide
        Ryan McKinley added a comment -

        but as a proof of concept i think it's pretty cool and worth pursuing

        Yes! This looks great chris. You discussed this a while back and I sort of ignored it as a real option (having messed with doclet configuration before)

        Thanks for doing the leg work to get something working,

        Show
        Ryan McKinley added a comment - but as a proof of concept i think it's pretty cool and worth pursuing Yes! This looks great chris. You discussed this a while back and I sort of ignored it as a real option (having messed with doclet configuration before) Thanks for doing the leg work to get something working,
        Hide
        Hoss Man added a comment -

        For the past few months, I've spent a few hours a week playing around with the javadoc doclet and taglet APIs. they are horrific to work with, but it cannot be denied: the ability to hook right into the javadoc process allows you to do some fairly powerful things.

        I've put together a new "SolrPluginDoclet" as well as a SolrInitParamTaglet and a SolrRunParamTaglet. The Doclet allows us to use the javadoc command to generate "user documentation" containing scaled down information about all of the identified "Solr Plugin" APIs along with information about how to use all of the implementations of those APIs found in the code base. The Taglets are used both by the SolrPluginDoclet when building the user docs, as well as by the standard javadoc doclet to output more structured information about initialization and request params that each type of plugin supports.

        There are still a lot of improvements to be made (see todo list below) before i would really consider this ready for end users – not the least of which is actually documenting all of our plugins using these structured javadoc tags – but as a proof of concept i think it's pretty cool and worth pursuing.

        You can see an example of what the generated docs currently look at the URLs below. Note the DemoPlugin and DemoChildPlugin files in the patch which show off the custom taglet...

        Traditional javadocs with new taglet added...

        Plugin docs for User...

        Todo List...

         *  - change output to first include a summary TOC of all classes by section,
         *    then list the actual info (standardize section header sizes)
         *  - make classes link to their full javadocs
         *  - add css+example for certain things to not display on plugin docs, just
         *    on javadocs.
         *  - make each plugin link to a wiki page with the same name (and some
         *    common prefix)
         *  - rename init.param:solr.initparam and run.param:solr.param
         *  - change param taglet to use inline @solr.type instead of @link in
         *    specific position
         *    (hmm... this is really hard .. skip for now)
         *  - make param taglets more resilient and log errors if @link for type isn't
         *    found in expected spot (might be helped by previous item if @link is
         *    replaced with @solr.type)
         *  - add a taglet for handlers to list their devault components,
         *    list params from components as well.
         *  - improve the CSS for everything so it's nice and pretty
        
        Show
        Hoss Man added a comment - For the past few months, I've spent a few hours a week playing around with the javadoc doclet and taglet APIs. they are horrific to work with, but it cannot be denied: the ability to hook right into the javadoc process allows you to do some fairly powerful things. I've put together a new "SolrPluginDoclet" as well as a SolrInitParamTaglet and a SolrRunParamTaglet. The Doclet allows us to use the javadoc command to generate "user documentation" containing scaled down information about all of the identified "Solr Plugin" APIs along with information about how to use all of the implementations of those APIs found in the code base. The Taglets are used both by the SolrPluginDoclet when building the user docs, as well as by the standard javadoc doclet to output more structured information about initialization and request params that each type of plugin supports. There are still a lot of improvements to be made (see todo list below) before i would really consider this ready for end users – not the least of which is actually documenting all of our plugins using these structured javadoc tags – but as a proof of concept i think it's pretty cool and worth pursuing. You can see an example of what the generated docs currently look at the URLs below. Note the DemoPlugin and DemoChildPlugin files in the patch which show off the custom taglet... Traditional javadocs with new taglet added... http://people.apache.org/~hossman/tmp/SOLR-555/org/apache/solr/DemoPlugin.html http://people.apache.org/~hossman/tmp/SOLR-555/org/apache/solr/DemoChildPlugin.html Plugin docs for User... http://people.apache.org/~hossman/tmp/SOLR-555/plugins.html http://people.apache.org/~hossman/tmp/SOLR-555/org.apache.solr.request.SolrRequestHandler.html Todo List... * - change output to first include a summary TOC of all classes by section, * then list the actual info (standardize section header sizes) * - make classes link to their full javadocs * - add css+example for certain things to not display on plugin docs, just * on javadocs. * - make each plugin link to a wiki page with the same name (and some * common prefix) * - rename init.param:solr.initparam and run.param:solr.param * - change param taglet to use inline @solr.type instead of @link in * specific position * (hmm... this is really hard .. skip for now) * - make param taglets more resilient and log errors if @link for type isn't * found in expected spot (might be helped by previous item if @link is * replaced with @solr.type) * - add a taglet for handlers to list their devault components, * list params from components as well. * - improve the CSS for everything so it's nice and pretty

          People

          • Assignee:
            Hoss Man
            Reporter:
            Hoss Man
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development