Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      There are some changes in 0.17 that require changes to pig code. Patch coming soon.

      1. hadoop17.jar
        4.16 MB
        Olga Natkovich
      2. PIG-198_v0.patch
        19 kB
        Olga Natkovich
      3. PIG-198_1_20080409.patch
        15 kB
        Arun C Murthy
      4. PIG-198_2_20080415.patch
        16 kB
        Arun C Murthy
      5. hadoop17.jar
        4.16 MB
        Nigel Daley
      6. PIG-198_3_20080416.patch
        18 kB
        Arun C Murthy
      7. PIG-198_4_20080429.patch
        19 kB
        Arun C Murthy
      8. PIG-198_5_20080509.patch
        19 kB
        Arun C Murthy
      9. hadoop17.jar
        4.16 MB
        Olga Natkovich

        Issue Links

          Activity

          Hide
          Olga Natkovich added a comment -

          hadoop 17 jar file

          Show
          Olga Natkovich added a comment - hadoop 17 jar file
          Hide
          Olga Natkovich added a comment -

          This patch is not ready but I will be on vacation and Arun will be taking this work over.

          With this patch all but to tests in TestEvalPipeline are passing. We suspect that this is due to the fact that pig does not correctly skip metadata files. (Also for debugging, the other tests cases in TestEvalPipeline are commented out. That of course needs to be reversed before the patch is committed.)

          The other problem is that the tests take twice as long and that issue also needs to be resolved before the patch is ready to go in

          Show
          Olga Natkovich added a comment - This patch is not ready but I will be on vacation and Arun will be taking this work over. With this patch all but to tests in TestEvalPipeline are passing. We suspect that this is due to the fact that pig does not correctly skip metadata files. (Also for debugging, the other tests cases in TestEvalPipeline are commented out. That of course needs to be reversed before the patch is committed.) The other problem is that the tests take twice as long and that issue also needs to be resolved before the patch is ready to go in
          Hide
          Arun C Murthy added a comment -

          Olga's patch along with the fix which makes TestEvalPipeline work again... I had to fix PigSlicer to ignore hadoop's meta files.

          All tests succeed and the whole test-suite took 20odd minutes to run, hence no regression from Pig with hadoop-0.16.

          Show
          Arun C Murthy added a comment - Olga's patch along with the fix which makes TestEvalPipeline work again... I had to fix PigSlicer to ignore hadoop's meta files. All tests succeed and the whole test-suite took 20odd minutes to run, hence no regression from Pig with hadoop-0.16.
          Hide
          Arun C Murthy added a comment -

          Need to regenerate this patch due to recent changes to trunk ...

          Show
          Arun C Murthy added a comment - Need to regenerate this patch due to recent changes to trunk ...
          Hide
          Arun C Murthy added a comment -

          Updated patch to reflect recent changes to trunk... this patch is blocked by HADOOP-3256.

          Show
          Arun C Murthy added a comment - Updated patch to reflect recent changes to trunk... this patch is blocked by HADOOP-3256 .
          Hide
          Nigel Daley added a comment -

          Updated Hadoop 0.17-dev jar

          Show
          Nigel Daley added a comment - Updated Hadoop 0.17-dev jar
          Hide
          Nigel Daley added a comment -

          This still needs some work. Pig doesn't work with HOD.

          HOD now sets fs.default.name to a full specified URI such as hdfs://foo.bar:8094 (before it just set for.bar:8094). In src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java, the fixUpDomain() method splits the string at the first colon and thinks everything after the first colon is the port. You can see that this assumption no longer works. I suspect that fixUpDomain should work with URI's that have and don't have a scheme (hdfs:// or file://).

          Show
          Nigel Daley added a comment - This still needs some work. Pig doesn't work with HOD. HOD now sets fs.default.name to a full specified URI such as hdfs://foo.bar:8094 (before it just set for.bar:8094). In src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java, the fixUpDomain() method splits the string at the first colon and thinks everything after the first colon is the port. You can see that this assumption no longer works. I suspect that fixUpDomain should work with URI's that have and don't have a scheme (hdfs:// or file:// ).
          Hide
          Arun C Murthy added a comment -

          Cancelling patch while I address concerns raised by Nigel about inter-operability with HOD...

          Show
          Arun C Murthy added a comment - Cancelling patch while I address concerns raised by Nigel about inter-operability with HOD...
          Hide
          Arun C Murthy added a comment -

          Updated patch to fix HExecutionEngine.fixUpDomain so that it handles heirarchical URIs ...

          Show
          Arun C Murthy added a comment - Updated patch to fix HExecutionEngine.fixUpDomain so that it handles heirarchical URIs ...
          Hide
          Arun C Murthy added a comment -

          Updated patch to incorporate PIG-151 for hadoop-0.17.0.

          Show
          Arun C Murthy added a comment - Updated patch to incorporate PIG-151 for hadoop-0.17.0.
          Hide
          Pi Song added a comment -

          Do we apply the concept of systemElement across all the element types (file and dir)?
          The patch seems a bit conflicting in that systemElement() is added at ElementDescriptor interface but the implementation only exists in HPath and LocalPath.

          BTW, can anyone give me a pointer to meta file in Hadoop 17 ?

          Show
          Pi Song added a comment - Do we apply the concept of systemElement across all the element types (file and dir)? The patch seems a bit conflicting in that systemElement() is added at ElementDescriptor interface but the implementation only exists in HPath and LocalPath. BTW, can anyone give me a pointer to meta file in Hadoop 17 ?
          Hide
          Olga Natkovich added a comment -

          Arun, could you update the patch to match the latest trunk, thanks.

          Show
          Olga Natkovich added a comment - Arun, could you update the patch to match the latest trunk, thanks.
          Hide
          Arun C Murthy added a comment -

          Updated patch to reflect recent changes to trunk... passes all unit tests locally.

          Show
          Arun C Murthy added a comment - Updated patch to reflect recent changes to trunk... passes all unit tests locally.
          Hide
          Olga Natkovich added a comment -

          Release version of hadoop 17 jar

          Show
          Olga Natkovich added a comment - Release version of hadoop 17 jar
          Hide
          Olga Natkovich added a comment -

          Now that Haoop 0.17 is officially released, I would like to commit this patch.

          The current status is that all unit and end-to-end tests pass. Unit tests are taking about 30-40% longer. We have spend a significant amount of time trying to figure out the cause but could not identify the problem. I think we should commit it as is and deal with the slowness separately.

          Could, one of the committers please review the patch, thanks.

          Show
          Olga Natkovich added a comment - Now that Haoop 0.17 is officially released, I would like to commit this patch. The current status is that all unit and end-to-end tests pass. Unit tests are taking about 30-40% longer. We have spend a significant amount of time trying to figure out the cause but could not identify the problem. I think we should commit it as is and deal with the slowness separately. Could, one of the committers please review the patch, thanks.
          Hide
          Pi Song added a comment -

          Except the HOD bit that I don't know much about message format, I didn't spot anything wrong.

          Show
          Pi Song added a comment - Except the HOD bit that I don't know much about message format, I didn't spot anything wrong.
          Hide
          Arun C Murthy added a comment -

          Thansk for the review Pi!

          Show
          Arun C Murthy added a comment - Thansk for the review Pi!
          Hide
          Olga Natkovich added a comment -

          Patch committed. Thanks Arun for contributing and Pi for reviewing.

          Note that by default, Pig now builds to work with hadoop 0.17. Also, I removed hadoop 0.15 jar.

          Show
          Olga Natkovich added a comment - Patch committed. Thanks Arun for contributing and Pi for reviewing. Note that by default, Pig now builds to work with hadoop 0.17. Also, I removed hadoop 0.15 jar.
          Hide
          Pi Song added a comment -

          I found a problem with this. The "bin/pig" script points to Hadoop-0.15 when we don't supply it anymore. Can someone with write access please fix it quickly?

          Show
          Pi Song added a comment - I found a problem with this. The "bin/pig" script points to Hadoop-0.15 when we don't supply it anymore. Can someone with write access please fix it quickly?
          Hide
          Alan Gates added a comment -

          Changed bin/pig to pick up hadoop17.jar by default.

          Show
          Alan Gates added a comment - Changed bin/pig to pick up hadoop17.jar by default.

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Olga Natkovich
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development