Hive
  1. Hive
  2. HIVE-1135

Use Anakia for version controlled documentation

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently the Hive Language Manual and many other critical pieces of documentation are on the Hive wiki.

      Right now we count on the author of a patch to follow up and add wiki entries. While we do a decent job with this, new features can be missed. Or using running older/newer branches can not locate relevant documentation for their branch.
      ..example of a perception I do not think we want to give off...
      http://dev.hubspot.com/bid/30170/Who-Loves-the-Magic-Undocumented-Hive-Mapjoin-This-Guy

      We should generate our documentation in the way hadoop & hbase does, inline using forest. I would like to take the lead on this, but we need a lot of consensus on doing this properly.

      1. hive-1135-3-patch.txt
        36 kB
        Edward Capriolo
      2. hive-1135-4-patch.txt
        36 kB
        Carl Steinbach
      3. hive-1135-5-patch.txt
        47 kB
        Edward Capriolo
      4. hive-1135-6-patch.txt
        49 kB
        Carl Steinbach
      5. hive-1335-1.patch.txt
        4 kB
        Edward Capriolo
      6. hive-1335-2.patch.txt
        41 kB
        Edward Capriolo
      7. jdom-1.1.jar
        150 kB
        Edward Capriolo
      8. jdom-1.1.LICENSE
        3 kB
        Edward Capriolo
      9. wtf.png
        115 kB
        Edward Capriolo

        Issue Links

          Activity

          Hide
          Jeff Hammerbacher added a comment -

          I'd suggest using something other than forrest for the official documentation. See https://issues.apache.org/jira/browse/AVRO-319 for some suggested replacements.

          Show
          Jeff Hammerbacher added a comment - I'd suggest using something other than forrest for the official documentation. See https://issues.apache.org/jira/browse/AVRO-319 for some suggested replacements.
          Hide
          Edward Capriolo added a comment -

          Jeff,

          I only knew the other projects used forest. Anakin sounds good just because it is an apache project. I do not have any experience with any of these. I was more focused on having documentation to match the branches.

          Show
          Edward Capriolo added a comment - Jeff, I only knew the other projects used forest. Anakin sounds good just because it is an apache project. I do not have any experience with any of these. I was more focused on having documentation to match the branches.
          Hide
          Edward Capriolo added a comment -

          Patch, docs command beings anakia from xdocs directory.

          Show
          Edward Capriolo added a comment - Patch, docs command beings anakia from xdocs directory.
          Hide
          Edward Capriolo added a comment -

          Edited the build.xml a bit to deal with some incorrect paths. Also added the HiveDataDefinitionStatements wiki page for reference. Is everyone ok with using anakia and this structure. I would like to get this cleaned up with the current docs in place. Then I will do some follow up tickets and add some wiki pages, let me know if everyone is happy with the overall xdocs->docs process.

          Show
          Edward Capriolo added a comment - Edited the build.xml a bit to deal with some incorrect paths. Also added the HiveDataDefinitionStatements wiki page for reference. Is everyone ok with using anakia and this structure. I would like to get this cleaned up with the current docs in place. Then I will do some follow up tickets and add some wiki pages, let me know if everyone is happy with the overall xdocs->docs process.
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/178/
          -----------------------------------------------------------

          Review request for Hive Developers.

          Summary
          -------

          Submitted on behalf of Ed Capriolo.

          This addresses bug hive-1135.
          http://issues.apache.org/jira/browse/hive-1135

          Diffs


          trunk/build.xml 953531
          trunk/hwi/web/set_processor.jsp 953531
          trunk/xdocs/docs/glossary.xml PRE-CREATION
          trunk/xdocs/docs/index.xml PRE-CREATION
          trunk/xdocs/docs/language_manual/data-manipulation-statements.xml PRE-CREATION
          trunk/xdocs/docs/language_manual/working_with_bucketed_tables.xml PRE-CREATION
          trunk/xdocs/docs/user/hwi.xml PRE-CREATION
          trunk/xdocs/site.css PRE-CREATION
          trunk/xdocs/stylesheets/project.xml PRE-CREATION
          trunk/xdocs/stylesheets/site.vsl PRE-CREATION
          trunk/xdocs/velocity.properties PRE-CREATION

          Diff: http://review.hbase.org/r/178/diff

          Testing
          -------

          Thanks,

          Carl

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/178/ ----------------------------------------------------------- Review request for Hive Developers. Summary ------- Submitted on behalf of Ed Capriolo. This addresses bug hive-1135. http://issues.apache.org/jira/browse/hive-1135 Diffs trunk/build.xml 953531 trunk/hwi/web/set_processor.jsp 953531 trunk/xdocs/docs/glossary.xml PRE-CREATION trunk/xdocs/docs/index.xml PRE-CREATION trunk/xdocs/docs/language_manual/data-manipulation-statements.xml PRE-CREATION trunk/xdocs/docs/language_manual/working_with_bucketed_tables.xml PRE-CREATION trunk/xdocs/docs/user/hwi.xml PRE-CREATION trunk/xdocs/site.css PRE-CREATION trunk/xdocs/stylesheets/project.xml PRE-CREATION trunk/xdocs/stylesheets/site.vsl PRE-CREATION trunk/xdocs/velocity.properties PRE-CREATION Diff: http://review.hbase.org/r/178/diff Testing ------- Thanks, Carl
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/178/#review217
          -----------------------------------------------------------

          Looks good overall.

          A couple suggestions:

          • Stash this stuff under docs/ instead of creating another top level directory (xdocs/)
          • Manage the jdom dependency with Ivy.
          • Limit the initial import to the convents of the Hive Language Manual. I think some things should actually stay on the wiki, but the language manual is definitely one of those things that we want to have in VCS.

          trunk/build.xml
          <http://review.hbase.org/r/178/#comment982>

          I think it would be good to use the full name ("anakia") instead of abbreviating it as "an".

          trunk/build.xml
          <http://review.hbase.org/r/178/#comment985>

          Please add a description so that this shows up in the output of 'ant -p'

          trunk/build.xml
          <http://review.hbase.org/r/178/#comment986>

          Can we rename this "check-for-anakia" and remove the "prepare-error" target, i.e. something similar to the way the checkstyle check is currently done?

          trunk/hwi/web/set_processor.jsp
          <http://review.hbase.org/r/178/#comment987>

          Did you mean to include this in the patch?

          trunk/xdocs/docs/glossary.xml
          <http://review.hbase.org/r/178/#comment988>

          This email address looks bogus. Perhaps this should be hive-dev@hadoop.apache.org? Or else leave it out entirely?

          Is the author property something that we can define in velocity.properties?

          trunk/xdocs/stylesheets/project.xml
          <http://review.hbase.org/r/178/#comment993>

          Let's try to stick with consistent indentation (2 characters).

          trunk/xdocs/stylesheets/site.vsl
          <http://review.hbase.org/r/178/#comment995>

          ASF header repeated twice.

          • Carl
          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/178/#review217 ----------------------------------------------------------- Looks good overall. A couple suggestions: Stash this stuff under docs/ instead of creating another top level directory (xdocs/) Manage the jdom dependency with Ivy. Limit the initial import to the convents of the Hive Language Manual. I think some things should actually stay on the wiki, but the language manual is definitely one of those things that we want to have in VCS. trunk/build.xml < http://review.hbase.org/r/178/#comment982 > I think it would be good to use the full name ("anakia") instead of abbreviating it as "an". trunk/build.xml < http://review.hbase.org/r/178/#comment985 > Please add a description so that this shows up in the output of 'ant -p' trunk/build.xml < http://review.hbase.org/r/178/#comment986 > Can we rename this "check-for-anakia" and remove the "prepare-error" target, i.e. something similar to the way the checkstyle check is currently done? trunk/hwi/web/set_processor.jsp < http://review.hbase.org/r/178/#comment987 > Did you mean to include this in the patch? trunk/xdocs/docs/glossary.xml < http://review.hbase.org/r/178/#comment988 > This email address looks bogus. Perhaps this should be hive-dev@hadoop.apache.org? Or else leave it out entirely? Is the author property something that we can define in velocity.properties? trunk/xdocs/stylesheets/project.xml < http://review.hbase.org/r/178/#comment993 > Let's try to stick with consistent indentation (2 characters). trunk/xdocs/stylesheets/site.vsl < http://review.hbase.org/r/178/#comment995 > ASF header repeated twice. Carl
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          On 2010-06-14 15:26:25, Carl Steinbach wrote:

          > Looks good overall.

          >

          > A couple suggestions:

          > * Stash this stuff under docs/ instead of creating another top level directory (xdocs/)

          > * Manage the jdom dependency with Ivy.

          > * Limit the initial import to the convents of the Hive Language Manual. I think some things should actually stay on the wiki, but the language manual is definitely one of those things that we want to have in VCS.

          • Carl

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/178/#review217
          -----------------------------------------------------------

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> On 2010-06-14 15:26:25, Carl Steinbach wrote: > Looks good overall. > > A couple suggestions: > * Stash this stuff under docs/ instead of creating another top level directory (xdocs/) > * Manage the jdom dependency with Ivy. > * Limit the initial import to the convents of the Hive Language Manual. I think some things should actually stay on the wiki, but the language manual is definitely one of those things that we want to have in VCS. Running the docs target causes a 'velocity.log' file to appear in the source root directory. This should probably be redirected to build/dist/docs. See http://velocity.apache.org/engine/releases/velocity-1.4/developer-guide.html#Velocity Configuration Keys and Values Carl ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/178/#review217 -----------------------------------------------------------
          Hide
          Edward Capriolo added a comment -

          FMI. What is the hbase review board?

          > Stash this stuff under docs/ instead of creating another top level directory (xdocs/)
          It seems like xdocs is convention, also did not want to step on whatever is in docs. I will see if they both can live in docs happily.

          >JDOM ivy.

          Right now most/all the stuff in /lib comes from ivy. We should open up a another ticket and convert the entire project to ivy.

          >velocity.log
          Yes my local version (next patch already fixes that)

          > * Limit the initial import to the convents of the Hive Language Manual. I think some things should actually stay on the wiki, but the language manual is definitely one of those things that we want to have in VCS.

          I agree the initial import should come from the Hive Language Manual only. To me wiki just screems, "I did not have time to write a full complete doc." Generalization coming: 99% of the things in the wiki should be in xdocs. Users only want one place for authoritative information. Wikis and xdoc will fall out of sync, confusion follows.

          Show
          Edward Capriolo added a comment - FMI. What is the hbase review board? > Stash this stuff under docs/ instead of creating another top level directory (xdocs/) It seems like xdocs is convention, also did not want to step on whatever is in docs. I will see if they both can live in docs happily. >JDOM ivy. Right now most/all the stuff in /lib comes from ivy. We should open up a another ticket and convert the entire project to ivy. >velocity.log Yes my local version (next patch already fixes that) > * Limit the initial import to the convents of the Hive Language Manual. I think some things should actually stay on the wiki, but the language manual is definitely one of those things that we want to have in VCS. I agree the initial import should come from the Hive Language Manual only. To me wiki just screems, "I did not have time to write a full complete doc." Generalization coming: 99% of the things in the wiki should be in xdocs. Users only want one place for authoritative information. Wikis and xdoc will fall out of sync, confusion follows.
          Hide
          Edward Capriolo added a comment -

          >JDOM ivy.

          Right now most/all the stuff in /lib DOES NOT comes from ivy. We should open up a another ticket and convert the entire project to ivy.

          Show
          Edward Capriolo added a comment - >JDOM ivy. Right now most/all the stuff in /lib DOES NOT comes from ivy. We should open up a another ticket and convert the entire project to ivy.
          Hide
          Edward Capriolo added a comment -

          Fixed all items.

          Show
          Edward Capriolo added a comment - Fixed all items.
          Hide
          HBase Review Board added a comment -

          Message from: "Carl Steinbach" <carl@cloudera.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/178/
          -----------------------------------------------------------

          (Updated 2010-06-16 16:51:58.636162)

          Review request for Hive Developers.

          Changes
          -------

          Updated diff.

          Summary
          -------

          Submitted on behalf of Ed Capriolo.

          This addresses bug hive-1135.
          http://issues.apache.org/jira/browse/hive-1135

          Diffs (updated)


          trunk/build.xml 955109
          trunk/docs/docs/index.xml PRE-CREATION
          trunk/docs/docs/language_manual/data-manipulation-statements.xml PRE-CREATION
          trunk/docs/docs/language_manual/working_with_bucketed_tables.xml PRE-CREATION
          trunk/docs/site.css PRE-CREATION
          trunk/docs/stylesheets/project.xml PRE-CREATION
          trunk/docs/stylesheets/site.vsl PRE-CREATION
          trunk/docs/velocity.properties PRE-CREATION

          Diff: http://review.hbase.org/r/178/diff

          Testing
          -------

          Thanks,

          Carl

          Show
          HBase Review Board added a comment - Message from: "Carl Steinbach" <carl@cloudera.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/178/ ----------------------------------------------------------- (Updated 2010-06-16 16:51:58.636162) Review request for Hive Developers. Changes ------- Updated diff. Summary ------- Submitted on behalf of Ed Capriolo. This addresses bug hive-1135. http://issues.apache.org/jira/browse/hive-1135 Diffs (updated) trunk/build.xml 955109 trunk/docs/docs/index.xml PRE-CREATION trunk/docs/docs/language_manual/data-manipulation-statements.xml PRE-CREATION trunk/docs/docs/language_manual/working_with_bucketed_tables.xml PRE-CREATION trunk/docs/site.css PRE-CREATION trunk/docs/stylesheets/project.xml PRE-CREATION trunk/docs/stylesheets/site.vsl PRE-CREATION trunk/docs/velocity.properties PRE-CREATION Diff: http://review.hbase.org/r/178/diff Testing ------- Thanks, Carl
          Hide
          Carl Steinbach added a comment -
          • Fixed broken image links in the stylesheet.
          • Moved docs/docs/ to docs/xdocs/
          • Added description to ant 'docs' target.
          Show
          Carl Steinbach added a comment - Fixed broken image links in the stylesheet. Moved docs/docs/ to docs/xdocs/ Added description to ant 'docs' target.
          Hide
          Carl Steinbach added a comment -

          I'm +1 on committing this (either Ed's last patch, or the one I just added that contains three very small tweaks).

          Show
          Carl Steinbach added a comment - I'm +1 on committing this (either Ed's last patch, or the one I just added that contains three very small tweaks).
          Hide
          Edward Capriolo added a comment -

          Cool on adding the logo....But something went wrong here. Unless I applied the patch from the left table looks wrong now. check the screen shot.

          Show
          Edward Capriolo added a comment - Cool on adding the logo....But something went wrong here. Unless I applied the patch from the left table looks wrong now. check the screen shot.
          Hide
          Edward Capriolo added a comment -

          Added the join page as well.

          Show
          Edward Capriolo added a comment - Added the join page as well.
          Hide
          Carl Steinbach added a comment -

          hive-1136-6-patch.txt:

          • Use Ivy to manage jdom dependency for docs target.
          Show
          Carl Steinbach added a comment - hive-1136-6-patch.txt: Use Ivy to manage jdom dependency for docs target.
          Hide
          Carl Steinbach added a comment -

          @Ed:

          • I updated the patch to use Ivy for managing the JDOM dependency.
          • I'm not sure how to fix the image alignment issue. Let's defer worrying about this for a followup ticket.
          • I will file a followup ticket that covers migrating the wiki docs over to version control. We need to do this in a consistent fashion in order to avoid missing any updates to the wiki that people make in the meantime. We also need to figure out how to push the generated docs to the website.
          • I'm +1 for committing this patch. Can you take care of that?
          Show
          Carl Steinbach added a comment - @Ed: I updated the patch to use Ivy for managing the JDOM dependency. I'm not sure how to fix the image alignment issue. Let's defer worrying about this for a followup ticket. I will file a followup ticket that covers migrating the wiki docs over to version control. We need to do this in a consistent fashion in order to avoid missing any updates to the wiki that people make in the meantime. We also need to figure out how to push the generated docs to the website. I'm +1 for committing this patch. Can you take care of that?
          Hide
          Edward Capriolo added a comment -

          Great on ivy.
          As for the wiki I think we should just put a node at the top of the page that says "Do not edit me. Edit xdocs instead." For the pages we have migrated. I want to do like a page every other day so it should be done soon enough. I actually have commit access but I usually leave the commits up to the experts. Also since I worked on this ticket I really should not be the commit person. Anyone else?

          Show
          Edward Capriolo added a comment - Great on ivy. As for the wiki I think we should just put a node at the top of the page that says "Do not edit me. Edit xdocs instead." For the pages we have migrated. I want to do like a page every other day so it should be done soon enough. I actually have commit access but I usually leave the commits up to the experts. Also since I worked on this ticket I really should not be the commit person. Anyone else?
          Hide
          Edward Capriolo added a comment -

          Bump:: I will fix the formatting later. Can we commit this we do not really need any unit tests here?

          Show
          Edward Capriolo added a comment - Bump:: I will fix the formatting later. Can we commit this we do not really need any unit tests here?
          Hide
          John Sichi added a comment -

          I should be able to get to this one early next week.

          Show
          John Sichi added a comment - I should be able to get to this one early next week.
          Hide
          John Sichi added a comment -

          Just a note: when supplying patches (whether from svn or git), make sure they can be applied with

          patch -p0 < hive-1135.?.patch

          For this one, I had to use patch -p1.

          Show
          John Sichi added a comment - Just a note: when supplying patches (whether from svn or git), make sure they can be applied with patch -p0 < hive-1135.?.patch For this one, I had to use patch -p1.
          Hide
          John Sichi added a comment -

          +1. Will commit if tests pass (I can't think why they wouldn't but hey).

          Show
          John Sichi added a comment - +1. Will commit if tests pass (I can't think why they wouldn't but hey).
          Hide
          John Sichi added a comment -

          Committed. Thanks Edward!

          Carl, could you open another JIRA issue for getting all the relevant docs moved from wiki to Anakia as part of 0.6 release, and also send out coordinating instructions on the mailing list so that people know about the change?

          Show
          John Sichi added a comment - Committed. Thanks Edward! Carl, could you open another JIRA issue for getting all the relevant docs moved from wiki to Anakia as part of 0.6 release, and also send out coordinating instructions on the mailing list so that people know about the change?
          Hide
          Edward Capriolo added a comment -

          Carl, thank you for the assist!

          Show
          Edward Capriolo added a comment - Carl, thank you for the assist!

            People

            • Assignee:
              Edward Capriolo
              Reporter:
              Edward Capriolo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development