HCatalog
  1. HCatalog
  2. HCATALOG-158

Update HAR support to work with Hadoop 205

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2, 0.3
    • Component/s: build
    • Labels:
      None

      Description

      Currently, HCat builds against hadoop_archive-0.3.1.jar which is statically included in lib directory. This is because Har did not used to be officially supported in the earlier versions of Hadoop and at the end we could not make it work with earlier versions and had to disable by default.

      Hadoop 205 provides native support for HAR which is included in hadoop_tools.jar. We need to switch the build to build against this jar picked from maven.

      1. HCAT-158_branch-0.2.patch
        2 kB
        Thomas Weise
      2. HCAT-158_trunk.patch
        2 kB
        Thomas Weise

        Issue Links

          Activity

          Hide
          Thomas Weise added a comment -

          Remove hadoop_archive and compile against hadoop_tools (20.205.0) without bundling it.

          Tests pass.

          Show
          Thomas Weise added a comment - Remove hadoop_archive and compile against hadoop_tools (20.205.0) without bundling it. Tests pass.
          Hide
          Ashutosh Chauhan added a comment -

          Marked hadoop jira will eliminate the need of hadoop-tools.jar altogether? So, we won't need that. But getting that fixed is not an requirement, right ?

          Show
          Ashutosh Chauhan added a comment - Marked hadoop jira will eliminate the need of hadoop-tools.jar altogether? So, we won't need that. But getting that fixed is not an requirement, right ?
          Hide
          Thomas Weise added a comment -

          Even with the next .20 hadoop release, we will still require separate hadoop_tools to compile, unless the Hive-Hadoop dependency is upgraded to .20.205.1. That by itself looks like a remote scenario and even then hadoop_tools won't hurt until there are code changes related to the HAR interface, at which time this added dependency can be dropped for an upgraded Hadoop version (.23+?).

          The immediate need is to eliminate bundled hadoop_archive as this breaks deployment.

          Show
          Thomas Weise added a comment - Even with the next .20 hadoop release, we will still require separate hadoop_tools to compile, unless the Hive-Hadoop dependency is upgraded to .20.205.1. That by itself looks like a remote scenario and even then hadoop_tools won't hurt until there are code changes related to the HAR interface, at which time this added dependency can be dropped for an upgraded Hadoop version (.23+?). The immediate need is to eliminate bundled hadoop_archive as this breaks deployment.
          Hide
          Ashutosh Chauhan added a comment -

          +1 its a good practice to get dep libs from ivy instead of being checked-in. Patch doesn't apply on trunk. Thomas, can you generate one for trunk as well. Currently running test on 0.2 branch. Will commit on 0.2 once tests pass.

          Show
          Ashutosh Chauhan added a comment - +1 its a good practice to get dep libs from ivy instead of being checked-in. Patch doesn't apply on trunk. Thomas, can you generate one for trunk as well. Currently running test on 0.2 branch. Will commit on 0.2 once tests pass.
          Hide
          Thomas Weise added a comment -

          Modified patch for trunk - build.xml changed.

          Show
          Thomas Weise added a comment - Modified patch for trunk - build.xml changed.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to 0.2 and trunk. Thanks, Thomas!

          Show
          Ashutosh Chauhan added a comment - Committed to 0.2 and trunk. Thanks, Thomas!
          Hide
          Alan Gates added a comment -

          Issue closed with 0.4 release.

          Show
          Alan Gates added a comment - Issue closed with 0.4 release.

            People

            • Assignee:
              Thomas Weise
              Reporter:
              Olga Natkovich
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development