HCatalog
  1. HCatalog
  2. HCATALOG-158

Update HAR support to work with Hadoop 205

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2, 0.3
    • Component/s: build
    • Labels:
      None

      Description

      Currently, HCat builds against hadoop_archive-0.3.1.jar which is statically included in lib directory. This is because Har did not used to be officially supported in the earlier versions of Hadoop and at the end we could not make it work with earlier versions and had to disable by default.

      Hadoop 205 provides native support for HAR which is included in hadoop_tools.jar. We need to switch the build to build against this jar picked from maven.

      1. HCAT-158_trunk.patch
        2 kB
        Thomas Weise
      2. HCAT-158_branch-0.2.patch
        2 kB
        Thomas Weise

        Issue Links

          Activity

          Olga Natkovich created issue -
          Thomas Weise made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-7810 [ HADOOP-7810 ]
          Thomas Weise made changes -
          Fix Version/s 0.3 [ 12317567 ]
          Component/s build [ 12315600 ]
          Hide
          Thomas Weise added a comment -

          Remove hadoop_archive and compile against hadoop_tools (20.205.0) without bundling it.

          Tests pass.

          Show
          Thomas Weise added a comment - Remove hadoop_archive and compile against hadoop_tools (20.205.0) without bundling it. Tests pass.
          Thomas Weise made changes -
          Attachment HCAT-158_branch-0.2.patch [ 12503396 ]
          Thomas Weise made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ashutosh Chauhan added a comment -

          Marked hadoop jira will eliminate the need of hadoop-tools.jar altogether? So, we won't need that. But getting that fixed is not an requirement, right ?

          Show
          Ashutosh Chauhan added a comment - Marked hadoop jira will eliminate the need of hadoop-tools.jar altogether? So, we won't need that. But getting that fixed is not an requirement, right ?
          Hide
          Thomas Weise added a comment -

          Even with the next .20 hadoop release, we will still require separate hadoop_tools to compile, unless the Hive-Hadoop dependency is upgraded to .20.205.1. That by itself looks like a remote scenario and even then hadoop_tools won't hurt until there are code changes related to the HAR interface, at which time this added dependency can be dropped for an upgraded Hadoop version (.23+?).

          The immediate need is to eliminate bundled hadoop_archive as this breaks deployment.

          Show
          Thomas Weise added a comment - Even with the next .20 hadoop release, we will still require separate hadoop_tools to compile, unless the Hive-Hadoop dependency is upgraded to .20.205.1. That by itself looks like a remote scenario and even then hadoop_tools won't hurt until there are code changes related to the HAR interface, at which time this added dependency can be dropped for an upgraded Hadoop version (.23+?). The immediate need is to eliminate bundled hadoop_archive as this breaks deployment.
          Hide
          Ashutosh Chauhan added a comment -

          +1 its a good practice to get dep libs from ivy instead of being checked-in. Patch doesn't apply on trunk. Thomas, can you generate one for trunk as well. Currently running test on 0.2 branch. Will commit on 0.2 once tests pass.

          Show
          Ashutosh Chauhan added a comment - +1 its a good practice to get dep libs from ivy instead of being checked-in. Patch doesn't apply on trunk. Thomas, can you generate one for trunk as well. Currently running test on 0.2 branch. Will commit on 0.2 once tests pass.
          Hide
          Thomas Weise added a comment -

          Modified patch for trunk - build.xml changed.

          Show
          Thomas Weise added a comment - Modified patch for trunk - build.xml changed.
          Thomas Weise made changes -
          Attachment HCAT-158_trunk.patch [ 12503463 ]
          Hide
          Ashutosh Chauhan added a comment -

          Committed to 0.2 and trunk. Thanks, Thomas!

          Show
          Ashutosh Chauhan added a comment - Committed to 0.2 and trunk. Thanks, Thomas!
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Alan Gates added a comment -

          Issue closed with 0.4 release.

          Show
          Alan Gates added a comment - Issue closed with 0.4 release.
          Alan Gates made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          1d 16h 48m 1 Thomas Weise 11/Nov/11 18:21
          Patch Available Patch Available Resolved Resolved
          8h 29m 1 Ashutosh Chauhan 12/Nov/11 02:51
          Resolved Resolved Closed Closed
          186d 22h 29m 1 Alan Gates 17/May/12 02:20

            People

            • Assignee:
              Thomas Weise
              Reporter:
              Olga Natkovich
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development