HCatalog
  1. HCatalog
  2. HCATALOG-14

Make it possible to build HCatalog using the HCatalog Git mirror

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There's a Git mirror of the HCatalog SVN repository available here: git://git.apache.org/hcatalog.git

      However, right now it's not possible to build HCatalog using the Git mirror because of the svn:external dependency on Hive:

      % ant
      Buildfile: /Users/carl/Work/repos/hcatalog/build.xml

      hive.jar:

      BUILD FAILED
      /Users/carl/Work/repos/hcatalog/build.xml:202: Basedir /Users/carl/Work/repos/hcatalog/hive/external does not exist

      Total time: 0 seconds

      1. hcat-git.patch
        1 kB
        Jakob Homan
      2. hcat-git-2.patch
        1 kB
        Jakob Homan

        Issue Links

          Activity

          Hide
          Jakob Homan added a comment -

          Attached patch adds hive as a git submodule, which gits it to build for me under git. Also adds a git-init ant task which would be necessary initially.

          Git submodules aren't exactly the same as svn externs, particularly in that you can't point them at specific revisions (per my reading), so this will work as long as we're tracking with trunk. Cutting release branches will be more difficult, but I think this patch should go in to save developers from svn hell.

          I cloned the hcat repository at https://github.com/jghoman/hcatalog and applied this patch, if anyone wants to test it. I had trouble applying it against the svn checkout as this patch tries to create hive/extern, which already exists. But this shouldn't cause a problem.

          I'm not a git expert, so there may be issues I'm not aware of, but this works for me.

          Show
          Jakob Homan added a comment - Attached patch adds hive as a git submodule, which gits it to build for me under git. Also adds a git-init ant task which would be necessary initially. Git submodules aren't exactly the same as svn externs, particularly in that you can't point them at specific revisions (per my reading), so this will work as long as we're tracking with trunk. Cutting release branches will be more difficult, but I think this patch should go in to save developers from svn hell. I cloned the hcat repository at https://github.com/jghoman/hcatalog and applied this patch, if anyone wants to test it. I had trouble applying it against the svn checkout as this patch tries to create hive/extern, which already exists. But this shouldn't cause a problem. I'm not a git expert, so there may be issues I'm not aware of, but this works for me.
          Hide
          Alan Gates added a comment -

          This won't work. In general we plan to have the extern tied to a particular version of Hive, rather than floating with trunk. Otherwise it will get too hard to keep hcat builds going and tests passing.

          Show
          Alan Gates added a comment - This won't work. In general we plan to have the extern tied to a particular version of Hive, rather than floating with trunk. Otherwise it will get too hard to keep hcat builds going and tests passing.
          Hide
          Jakob Homan added a comment -

          Actually, I was wrong. Git tracks specific revisions, which is want we want in the long term but not in the short, since pointing at the 7 branch breaks the compile. Here's a patch that does this. Unfortunately, there's no way to apply it via subversion since git sets a property on the external folder (the specific revision to point to, currently at 6e7d9afd4b6f2118fb173aa516f9e53ae8f6f2f6, the 7 branch).

          I can't see a way to set this git property via svn. Anyone know? In the meantime, it works for me to apply this against my working branches, which lets me work on hcat from git. I can then un-apply this patch before generating the actual patch I'm aiming for.

          In the long term, since git doesn't support externals, it's probably best to focus on removing the hive extern. I'll ask on Stack Overflow about the svn property, in case I'm missing something.

          Show
          Jakob Homan added a comment - Actually, I was wrong. Git tracks specific revisions, which is want we want in the long term but not in the short, since pointing at the 7 branch breaks the compile. Here's a patch that does this. Unfortunately, there's no way to apply it via subversion since git sets a property on the external folder (the specific revision to point to, currently at 6e7d9afd4b6f2118fb173aa516f9e53ae8f6f2f6, the 7 branch). I can't see a way to set this git property via svn. Anyone know? In the meantime, it works for me to apply this against my working branches, which lets me work on hcat from git. I can then un-apply this patch before generating the actual patch I'm aiming for. In the long term, since git doesn't support externals, it's probably best to focus on removing the hive extern. I'll ask on Stack Overflow about the svn property, in case I'm missing something.
          Hide
          Jakob Homan added a comment -

          Since there's been no response to my query about making svn do what we need it to do (http://bit.ly/jb1TF0), I was wondering why it's necessary to have the extern to Hive included at all, rather than just including a jar to Hive? Hive's now published to Maven: https://repository.apache.org/content/repositories/snapshots/org/apache/hive/, so snapshots should be available. Are developers expecting to create patches against Hive and HCatalog at the same time? What's the expected timeframe for removing the Hive external?

          Show
          Jakob Homan added a comment - Since there's been no response to my query about making svn do what we need it to do ( http://bit.ly/jb1TF0 ), I was wondering why it's necessary to have the extern to Hive included at all, rather than just including a jar to Hive? Hive's now published to Maven: https://repository.apache.org/content/repositories/snapshots/org/apache/hive/ , so snapshots should be available. Are developers expecting to create patches against Hive and HCatalog at the same time? What's the expected timeframe for removing the Hive external?
          Hide
          Alan Gates added a comment -

          It doesn't appear that this is automatically updated, since the last update is May 17th. I'm not clear who has the power to update these snapshot releases. When there's a new feature in Hive's trunk that we need how do we get a new snapshot uploaded?

          In general, I thought Carl had a solution for git users where a target would be added to build that would read the hive version needed from SVN and then check out the proper git reversion. This depended on git knowing the SVN revision numbers so you could check out properly. Carl, did you get a chance to explore whether this would work?

          Show
          Alan Gates added a comment - It doesn't appear that this is automatically updated, since the last update is May 17th. I'm not clear who has the power to update these snapshot releases. When there's a new feature in Hive's trunk that we need how do we get a new snapshot uploaded? In general, I thought Carl had a solution for git users where a target would be added to build that would read the hive version needed from SVN and then check out the proper git reversion. This depended on git knowing the SVN revision numbers so you could check out properly. Carl, did you get a chance to explore whether this would work?
          Hide
          Carl Steinbach added a comment - - edited

          When there's a new feature in Hive's trunk that we need how do we get a new snapshot uploaded?

          I modified Hive's Hudson job to automatically run the 'maven-build' and 'maven-publish' tasks, which should hopefully push artifacts to the snapshot repository on every successful build. I'll let you know if I can get this to work. Assuming it does, will this satisfy HCatalog's requirements? Besides the Hive JARs, do you need access to anything else like configuration files or shell scripts?

          PS: Can you ask the INFRA folks to turn on wiki markup for the HCatalog JIRA group?

          Show
          Carl Steinbach added a comment - - edited When there's a new feature in Hive's trunk that we need how do we get a new snapshot uploaded? I modified Hive's Hudson job to automatically run the 'maven-build' and 'maven-publish' tasks, which should hopefully push artifacts to the snapshot repository on every successful build. I'll let you know if I can get this to work. Assuming it does, will this satisfy HCatalog's requirements? Besides the Hive JARs, do you need access to anything else like configuration files or shell scripts? PS: Can you ask the INFRA folks to turn on wiki markup for the HCatalog JIRA group?
          Hide
          Carl Steinbach added a comment -

          The Hive Hudson jobs are now publishing SNAPSHOT artifacts to the apache repository on every successful build:

          https://repository.apache.org/content/repositories/snapshots/org/apache/hive/

          Show
          Carl Steinbach added a comment - The Hive Hudson jobs are now publishing SNAPSHOT artifacts to the apache repository on every successful build: https://repository.apache.org/content/repositories/snapshots/org/apache/hive/
          Hide
          Jakob Homan added a comment -

          Now that Hive is no longer externed and is being brought in via ivy/maven, building from git works just fine. Resolving.

          Show
          Jakob Homan added a comment - Now that Hive is no longer externed and is being brought in via ivy/maven, building from git works just fine. Resolving.

            People

            • Assignee:
              Unassigned
              Reporter:
              Carl Steinbach
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development