Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.11.0
    • Component/s: HCatalog
    • Labels:
      None

      Description

      The HCatalog code needs to be moved into Hive.

        Activity

        Hide
        Alan Gates added a comment -

        We definitely want them to be one release with one artifact containing all of Hive (which now also includes the HCat jars and executables). The only case I could see for a separate HCat release is if there's a need to do a 0.5.1 release to address bugs in 0.5.0.

        Show
        Alan Gates added a comment - We definitely want them to be one release with one artifact containing all of Hive (which now also includes the HCat jars and executables). The only case I could see for a separate HCat release is if there's a need to do a 0.5.1 release to address bugs in 0.5.0.
        Hide
        Carl Steinbach added a comment -

        The plan of record is for HCatalog to be one of the artifacts contained in the Hive release.

        Show
        Carl Steinbach added a comment - The plan of record is for HCatalog to be one of the artifacts contained in the Hive release.
        Hide
        Travis Crawford added a comment -

        Hey all - great work with getting the code into the Hive repo so far! As we make progress here I'd like to get everyone on the same page about what our final product is. That is, when someone builds a Hive package (ant clean package) how does HCatalog fit in? Do we continue to release two artifacts (separate Hive & HCatalog releases) or just one Hive artifact that also contains HCatalog? Once we're all on the same page here (I strongly prefer one Hive artifact that also contains HCatalog) we can figure out what the merged release looks like.

        THoughts?

        Show
        Travis Crawford added a comment - Hey all - great work with getting the code into the Hive repo so far! As we make progress here I'd like to get everyone on the same page about what our final product is. That is, when someone builds a Hive package ( ant clean package ) how does HCatalog fit in? Do we continue to release two artifacts (separate Hive & HCatalog releases) or just one Hive artifact that also contains HCatalog? Once we're all on the same page here (I strongly prefer one Hive artifact that also contains HCatalog) we can figure out what the merged release looks like. THoughts?
        Hide
        Sushanth Sowmyan added a comment -

        (As a checkpoint, as of revision 651fd3acebd1e54a203824e9e8a0a8d6add21875, hcat trunk from inside hive compiles cleanly and all tests pass successfully)

        Show
        Sushanth Sowmyan added a comment - (As a checkpoint, as of revision 651fd3acebd1e54a203824e9e8a0a8d6add21875, hcat trunk from inside hive compiles cleanly and all tests pass successfully)
        Hide
        Alan Gates added a comment -

        Taken from hive dev list:

        What changes will be needed to make HCatalog's documentation set buildable
        in its Hive location? Or will all HCatalog docs need to be converted to
        wikidocs?

        (Tech note: HCatalog has xdocs built with Forrest, unlike the dearly
        departed Hive xdocs which were built with Anakia.)

        – Lefty

        This was covered in the original proposal for merging HCatalog into Hive:

        5) Website
        We will need to integrate HCatalog's website with Hive's. This should be
        easy except for the documentation. HCat uses forrest for docs, Hive uses
        wiki. We will need to put links under 'Documentation' for older versions
        of HCat docs so users can find them. As far as how docs are handled for
        the next version of HCatalog, I think that depends on the answer to
        question 7 (next release of HCat), but I propose that HCat needs to conform
        to the way Hive does docs on wiki. Though I would strongly encourage the
        HCat docs to be version specific (that is, have a set of wiki pages for
        each version). incubator.apache.org/hcatalog should be changed to
        forward to hive.apache.org.

        So in the initial checkin they will be built still with Forrest. We need to move them into the wiki to conform to the way Hive does docs.

        Show
        Alan Gates added a comment - Taken from hive dev list: What changes will be needed to make HCatalog's documentation set buildable in its Hive location? Or will all HCatalog docs need to be converted to wikidocs? (Tech note: HCatalog has xdocs built with Forrest, unlike the dearly departed Hive xdocs which were built with Anakia.) – Lefty This was covered in the original proposal for merging HCatalog into Hive: 5) Website We will need to integrate HCatalog's website with Hive's. This should be easy except for the documentation. HCat uses forrest for docs, Hive uses wiki. We will need to put links under 'Documentation' for older versions of HCat docs so users can find them. As far as how docs are handled for the next version of HCatalog, I think that depends on the answer to question 7 (next release of HCat), but I propose that HCat needs to conform to the way Hive does docs on wiki. Though I would strongly encourage the HCat docs to be version specific (that is, have a set of wiki pages for each version). incubator.apache.org/hcatalog should be changed to forward to hive.apache.org. So in the initial checkin they will be built still with Forrest. We need to move them into the wiki to conform to the way Hive does docs.
        Hide
        Alan Gates added a comment -

        Mithun pointed out that I'm only proposing how to move trunk here, not the branches, tags, and site sections. These will need to be moved as well. I don't think it makes any sense to merge these into Hive, Especially since we want to be able to release a 0.5.1 in the future if we need to, or if users wish to continue to build off of the existing branches without taking in the merge into Hive.

        So, I modify my proposal as follows: I'll move all of the existing HCatalog code (trunk, branches, tags, site) into hive/hcatalog/hcatalog-historical. I'll then move/re-arrange ONLY the trunk section of the code, as outlined above. So when I'm done we'll have the following:

        hive/
          hcatalog/
            build.xml
            core/
            hcatalog-pig-adapter/
            ...
            hcatalog-historical/
              branches/
              site/
              tags/
        
        Show
        Alan Gates added a comment - Mithun pointed out that I'm only proposing how to move trunk here, not the branches, tags, and site sections. These will need to be moved as well. I don't think it makes any sense to merge these into Hive, Especially since we want to be able to release a 0.5.1 in the future if we need to, or if users wish to continue to build off of the existing branches without taking in the merge into Hive. So, I modify my proposal as follows: I'll move all of the existing HCatalog code (trunk, branches, tags, site) into hive/hcatalog/hcatalog-historical. I'll then move/re-arrange ONLY the trunk section of the code, as outlined above. So when I'm done we'll have the following: hive/ hcatalog/ build.xml core/ hcatalog-pig-adapter/ ... hcatalog-historical/ branches/ site/ tags/
        Hide
        Ashutosh Chauhan added a comment -

        +1 Plan looks reasonable to me.

        Show
        Ashutosh Chauhan added a comment - +1 Plan looks reasonable to me.
        Hide
        Alan Gates added a comment -

        The HCatalog code needs to be moved into Hive from the incubator. This will be done by an 'svn mv' command in order to preserve history. I propose to do this as follows:

        1. I will move all of the hcat code as is under the hcatalog stub directory created in HIVE-4145. I will put it into a sub-directory to avoid clashes with files names that have already been created (such as build.xml). The command for this will look like: 'svn mv https://svn.apache.org/repos/asf//incubator/hcatalog/trunk https://svn.apache.org/repos/asf//hive/trunk/hcatalog/hcat-code-dump' This is a call on the repo and cannot be uploaded as a patch to JIRA.
        2. I can then deal with the files in that directory in separate commits.
          1. DISCLAIMER.txt can be removed as HCat is no longer incubating
          2. KEYS, LICENSE.txt, and NOTICE.TXT can be merged with the top level Hive files, as HCat no longer needs to track these entities separately.
          3. I propose we put CHANGES.txt, README.txt, and RELEASE_NOTES.txt in an newly created 'pre-hive-archive' directory. Going forward HCat will no longer need separate versions of these files. But for the sake of history it would be nice to keep them around.
          4. build.xml will be merged with the build.xml that was created in HIVE-4145
          5. The other directories and files will be moved to the top level.
          6. The Java code in these directories will be refactored into org.apache.hive.hcatalog and stub classes for org.apache.hcatalog classes will be created. This will only be done for classes available through public APIs.

        I further propose that for the duration of these operations only, the hcatalog directory will be CTR (commit then review) so that I do not have to post patches and wait for a review for each of these file move/merge operations. I will have JIRAs for each of these operations so that they are trackable in the future.

        I am proposing this double hop (move to hcat-code-dump and then from there) for two reasons. One, trying to do this in one patch is not doable based on how svn mv works. Two, it will be much easier to track and understand what is happening (both now and for users in the future reading the svn history) if it is done in a set of discrete steps rather than in one mega-move that also removes, moves, and merges files and directories.

        Show
        Alan Gates added a comment - The HCatalog code needs to be moved into Hive from the incubator. This will be done by an 'svn mv' command in order to preserve history. I propose to do this as follows: I will move all of the hcat code as is under the hcatalog stub directory created in HIVE-4145 . I will put it into a sub-directory to avoid clashes with files names that have already been created (such as build.xml). The command for this will look like: 'svn mv https://svn.apache.org/repos/asf//incubator/hcatalog/trunk https://svn.apache.org/repos/asf//hive/trunk/hcatalog/hcat-code-dump ' This is a call on the repo and cannot be uploaded as a patch to JIRA. I can then deal with the files in that directory in separate commits. DISCLAIMER.txt can be removed as HCat is no longer incubating KEYS, LICENSE.txt, and NOTICE.TXT can be merged with the top level Hive files, as HCat no longer needs to track these entities separately. I propose we put CHANGES.txt, README.txt, and RELEASE_NOTES.txt in an newly created 'pre-hive-archive' directory. Going forward HCat will no longer need separate versions of these files. But for the sake of history it would be nice to keep them around. build.xml will be merged with the build.xml that was created in HIVE-4145 The other directories and files will be moved to the top level. The Java code in these directories will be refactored into org.apache.hive.hcatalog and stub classes for org.apache.hcatalog classes will be created. This will only be done for classes available through public APIs. I further propose that for the duration of these operations only, the hcatalog directory will be CTR (commit then review) so that I do not have to post patches and wait for a review for each of these file move/merge operations. I will have JIRAs for each of these operations so that they are trackable in the future. I am proposing this double hop (move to hcat-code-dump and then from there) for two reasons. One, trying to do this in one patch is not doable based on how svn mv works. Two, it will be much easier to track and understand what is happening (both now and for users in the future reading the svn history) if it is done in a set of discrete steps rather than in one mega-move that also removes, moves, and merges files and directories.

          People

          • Assignee:
            Alan Gates
            Reporter:
            Alan Gates
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development