Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5
    • Fix Version/s: 0.5
    • Component/s: None
    • Labels:
      None

      Description

      We fixed hcatalog for hadoop 2 in HCATALOG-375. However, it is broken with recent changes. Plus, the new component webhcat never works with hadoop 2. We need to fix it.

      1. HCATALOG-624-1.patch
        41 kB
        Daniel Dai
      2. HCATALOG-624-2.patch
        39 kB
        Daniel Dai
      3. HCATALOG-624-3.patch
        40 kB
        Daniel Dai
      4. HCATALOG-624-4.patch
        36 kB
        Daniel Dai
      5. HCATALOG-624-5.patch
        0.7 kB
        Daniel Dai

        Issue Links

          Activity

          Hide
          Daniel Dai added a comment -

          HCATALOG-624-5.patch committed to trunk.

          Show
          Daniel Dai added a comment - HCATALOG-624 -5.patch committed to trunk.
          Hide
          Daniel Dai added a comment -

          See compilation issue with the commit. Attach HCATALOG-624-5.patch to fix it.

          Show
          Daniel Dai added a comment - See compilation issue with the commit. Attach HCATALOG-624 -5.patch to fix it.
          Hide
          Daniel Dai added a comment -

          Patch committed to trunk.

          Open HCATALOG-638 to track hbase hadoop 2 maven artifacts.

          Show
          Daniel Dai added a comment - Patch committed to trunk. Open HCATALOG-638 to track hbase hadoop 2 maven artifacts.
          Hide
          Sushanth Sowmyan added a comment -

          +1 on 4.patch.

          Thanks, Daniel!

          Show
          Sushanth Sowmyan added a comment - +1 on 4.patch. Thanks, Daniel!
          Hide
          Daniel Dai added a comment -

          Separate the patch into HCATALOG-624-4.patch and HCATALOG-637.

          Show
          Daniel Dai added a comment - Separate the patch into HCATALOG-624 -4.patch and HCATALOG-637 .
          Hide
          Sushanth Sowmyan added a comment -

          Okay, not a problem, I guess we can wait on the hadoop2 artifacts for hbase, and in the meanwhile, build them ourselves if needed.

          Could you please separate out the pig version upgrade as a separate patch and make that a blocker for this? That makes it easier for us in the future to track patches.

          Thanks!

          Show
          Sushanth Sowmyan added a comment - Okay, not a problem, I guess we can wait on the hadoop2 artifacts for hbase, and in the meanwhile, build them ourselves if needed. Could you please separate out the pig version upgrade as a separate patch and make that a blocker for this? That makes it easier for us in the future to track patches. Thanks!
          Hide
          Daniel Dai added a comment -

          a) After I got into testing, I didn't read the jira comments till after I'd run them all and had the hbase tests fail. This is entirely my fault for not reading your instructions, but something I think people will hit when they download hcat and try running tests. Do you know if mvn artifacts for hadoop2 going to be published by hbase?

          There is no hbase h2 artifacts yet. They don't have a ticket for that either, I will post one if they create one

          b) I see you changed some of the tests with pig schemas with parenthesis to not have the parenthesis any more, along with changing the pig dependency version. Should that be a separate patch by itself? I think it might make sense to separate 23-migration bugs with pig-upgrade and schema change bugs. Thoughts?

          This is due to pig version upgrade (0.8->0.10). Things are intermingled here. We upgrade Pig because only Pig 0.10 has h2 artifacts, and that's part of this ticket. I can create a ticket as part of this if needed.

          Show
          Daniel Dai added a comment - a) After I got into testing, I didn't read the jira comments till after I'd run them all and had the hbase tests fail. This is entirely my fault for not reading your instructions, but something I think people will hit when they download hcat and try running tests. Do you know if mvn artifacts for hadoop2 going to be published by hbase? There is no hbase h2 artifacts yet. They don't have a ticket for that either, I will post one if they create one b) I see you changed some of the tests with pig schemas with parenthesis to not have the parenthesis any more, along with changing the pig dependency version. Should that be a separate patch by itself? I think it might make sense to separate 23-migration bugs with pig-upgrade and schema change bugs. Thoughts? This is due to pig version upgrade (0.8->0.10). Things are intermingled here. We upgrade Pig because only Pig 0.10 has h2 artifacts, and that's part of this ticket. I can create a ticket as part of this if needed.
          Hide
          Sushanth Sowmyan added a comment -

          Read through to understand, and tested building and testing with hadoop23 and hadoop20, I checked the 624-2 patch against branch 0.5. I've not looked at your trunk patch yet, but I expect it to be okay, will do a quick verify before I +1.

          Basically, looks good, and tests succeed. A couple of comments:

          a) After I got into testing, I didn't read the jira comments till after I'd run them all and had the hbase tests fail. This is entirely my fault for not reading your instructions, but something I think people will hit when they download hcat and try running tests. Do you know if mvn artifacts for hadoop2 going to be published by hbase?
          b) I see you changed some of the tests with pig schemas with parenthesis to not have the parenthesis any more, along with changing the pig dependency version. Should that be a separate patch by itself? I think it might make sense to separate 23-migration bugs with pig-upgrade and schema change bugs. Thoughts?

          Show
          Sushanth Sowmyan added a comment - Read through to understand, and tested building and testing with hadoop23 and hadoop20, I checked the 624-2 patch against branch 0.5. I've not looked at your trunk patch yet, but I expect it to be okay, will do a quick verify before I +1. Basically, looks good, and tests succeed. A couple of comments: a) After I got into testing, I didn't read the jira comments till after I'd run them all and had the hbase tests fail. This is entirely my fault for not reading your instructions, but something I think people will hit when they download hcat and try running tests. Do you know if mvn artifacts for hadoop2 going to be published by hbase? b) I see you changed some of the tests with pig schemas with parenthesis to not have the parenthesis any more, along with changing the pig dependency version. Should that be a separate patch by itself? I think it might make sense to separate 23-migration bugs with pig-upgrade and schema change bugs. Thoughts?
          Hide
          Daniel Dai added a comment -

          Attach HCATALOG-624-3.patch to resync with trunk.

          Show
          Daniel Dai added a comment - Attach HCATALOG-624 -3.patch to resync with trunk.
          Hide
          Daniel Dai added a comment -

          Rework the shims for webhcat, attach HCATALOG-624-2.patch.

          Show
          Daniel Dai added a comment - Rework the shims for webhcat, attach HCATALOG-624 -2.patch.
          Hide
          Daniel Dai added a comment - - edited

          Attached patch fix all unit tests. However, there is no hbase mvn artifacts for hadoop 2. We need to build hbase by ourself with the flag "-Dhadoop.profile=2.0".

          To run hcat tests for hadoop 2, use the following command:
          ant -Dmvn.hadoop.profile=hadoop23 test

          Show
          Daniel Dai added a comment - - edited Attached patch fix all unit tests. However, there is no hbase mvn artifacts for hadoop 2. We need to build hbase by ourself with the flag "-Dhadoop.profile=2.0". To run hcat tests for hadoop 2, use the following command: ant -Dmvn.hadoop.profile=hadoop23 test

            People

            • Assignee:
              Daniel Dai
              Reporter:
              Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development