Pig
  1. Pig
  2. PIG-2125

Make Pig work with hadoop .NEXT

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.1, 0.10.0
    • Fix Version/s: 0.9.2, 0.10.0
    • Component/s: impl
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We need to make Pig work with hadoop .NEXT, the svn branch currently is: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279

      1. PIG-2125-1.patch
        20 kB
        Daniel Dai
      2. PIG-2125-2.patch
        169 kB
        Daniel Dai
      3. PIG-2125-3.patch
        0.8 kB
        Daniel Dai
      4. PIG-2125-4.patch
        183 kB
        Daniel Dai
      5. PIG-2125-5.patch
        3 kB
        Daniel Dai
      6. ContextFactory.java
        9 kB
        Daniel Dai
      7. PIG-2125-6.patch
        72 kB
        Daniel Dai
      8. e2e-hadoop23.patch
        0.5 kB
        Daniel Dai
      9. PIG-2125-7.patch
        74 kB
        Daniel Dai
      10. PIG-2125-7_0.9.patch
        72 kB
        Daniel Dai
      11. PIG-2125-buildxml-0.9.patch
        3 kB
        Thomas Weise
      12. PIG-2125-commitJob.patch
        3 kB
        Daniel Dai
      13. PIG-2125-zebra.patch
        2 kB
        Daniel Dai
      14. PIG-2125-ivy-0.9-3.patch
        4 kB
        Thomas Weise
      15. PIG-2125-8.patch
        84 kB
        Daniel Dai
      16. PIG-2125-9.patch
        84 kB
        Daniel Dai
      17. PIG-2125-9_0.9.patch
        83 kB
        Daniel Dai
      18. PIG-2125-10.patch
        84 kB
        Daniel Dai
      19. PIG-2125-10_0.9.patch
        84 kB
        Daniel Dai

        Issue Links

          Activity

          Daniel Dai created issue -
          Daniel Dai made changes -
          Field Original Value New Value
          Description We need to make Pig work with hadoop .NEXT, the svn branch currently is: https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
          Hide
          Daniel Dai added a comment -

          Attach initial patch PIG-2125-1.patch solving some issues we find. There are still areas under working, such as illustrator, MiniCluster, MergedJoin.

          Show
          Daniel Dai added a comment - Attach initial patch PIG-2125 -1.patch solving some issues we find. There are still areas under working, such as illustrator, MiniCluster, MergedJoin.
          Daniel Dai made changes -
          Attachment PIG-2125-1.patch [ 12482695 ]
          Hide
          Dmitriy V. Ryaboy added a comment -

          are you guys planning on skipping 22 altogether?

          Show
          Dmitriy V. Ryaboy added a comment - are you guys planning on skipping 22 altogether?
          Hide
          Olga Natkovich added a comment -

          We don't have any current plans to use 22 at Yahoo or for Yahoo engineers to do the port. But of course a patch is welcome from anybody interested in taking this project on.

          Show
          Olga Natkovich added a comment - We don't have any current plans to use 22 at Yahoo or for Yahoo engineers to do the port. But of course a patch is welcome from anybody interested in taking this project on.
          Hide
          Daniel Dai added a comment -

          PIG-2125-2.patch tested against MR-279 branch. All Pig end-to-end test pass. Unit test still fail. Zebra test still have a few failures.

          The patch borrow some ideas from PIG-924. However, a dynamic shims layer is not possible due to some static dependencies. Here we use a static shims layer, so for different hadoop version, we need to recompile.

          Since hadoop.Next maven artifacts is not available, so we need to simulate it by copying all libraries to lib.

          Show
          Daniel Dai added a comment - PIG-2125 -2.patch tested against MR-279 branch. All Pig end-to-end test pass. Unit test still fail. Zebra test still have a few failures. The patch borrow some ideas from PIG-924 . However, a dynamic shims layer is not possible due to some static dependencies. Here we use a static shims layer, so for different hadoop version, we need to recompile. Since hadoop.Next maven artifacts is not available, so we need to simulate it by copying all libraries to lib.
          Daniel Dai made changes -
          Attachment PIG-2125-2.patch [ 12484575 ]
          Hide
          Daniel Dai added a comment -

          PIG-2125-3.patch is additional patch to make Zebra work.

          Show
          Daniel Dai added a comment - PIG-2125 -3.patch is additional patch to make Zebra work.
          Daniel Dai made changes -
          Attachment PIG-2125-3.patch [ 12484903 ]
          Hide
          Alan Gates added a comment -

          In general the patch looks good. I have a few questions/comments.

          1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA.
          2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed.
          3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.
          Show
          Alan Gates added a comment - In general the patch looks good. I have a few questions/comments. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.
          Hide
          Daniel Dai added a comment -

          1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA.

          They need to copy all hadoop 23 jars and dependencies into lib. This step should go away once hadoop 23 publish maven artifacts

          2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed.

          Sure. I will add

          3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better.

          How about org.apache.pig.backend.hadoop.shims?

          Show
          Daniel Dai added a comment - 1. How does one build a 0.23 compatible version of Pig? Do you have to download the 0.23 jar yourself? It looked like if you did this, build.xml would manage the rest properly. The answer to this question will need to be added to the release notes field of this JIRA. They need to copy all hadoop 23 jars and dependencies into lib. This step should go away once hadoop 23 publish maven artifacts 2. Javadocs explaining what the classes do would be very helpful. PigMapReduce, PigMapBase, HadoopShims all could use class level comments. HadoopShims in particular should have extensive comments on how the shims layer works and what needs to be shimmed. Sure. I will add 3. I wonder if HadoopShims should go in a separate package. I know it's just one class, but it performs a unique function. This would also allow many of the design and build comments I referenced in the last point to be covered in a package level doc. I don't feel strongly about this, so feel free to keep it where it is if you think that's better. How about org.apache.pig.backend.hadoop.shims?
          Hide
          Daniel Dai added a comment -

          PIG-2125-4.patch merge PIG-2125-2.patch and PIG-2125-3.PATCH, address comments from Alan, fix unit test failures and test-patch failures.

          Show
          Daniel Dai added a comment - PIG-2125 -4.patch merge PIG-2125 -2.patch and PIG-2125 -3.PATCH, address comments from Alan, fix unit test failures and test-patch failures.
          Daniel Dai made changes -
          Attachment PIG-2125-4.patch [ 12486790 ]
          Daniel Dai made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed]
          Hide
          Daniel Dai added a comment -

          PIG-2125-4.patch committed to both trunk and 0.9 branch. This is the first phase work (mapreduce and local mode end-to-end), I will continue work on the next phase (unit test).

          Show
          Daniel Dai added a comment - PIG-2125 -4.patch committed to both trunk and 0.9 branch. This is the first phase work (mapreduce and local mode end-to-end), I will continue work on the next phase (unit test).
          Hide
          Daniel Dai added a comment -

          There are some piggybank and zebra test failures, attach PIG-2125-5.patch for the fix.

          Show
          Daniel Dai added a comment - There are some piggybank and zebra test failures, attach PIG-2125 -5.patch for the fix.
          Daniel Dai made changes -
          Attachment PIG-2125-5.patch [ 12487240 ]
          Hide
          Richard Ding added a comment -

          +1

          Show
          Richard Ding added a comment - +1
          Hide
          Daniel Dai added a comment -

          PIG-2125-5.patch committed to both trunk and 0.9 branch.

          Show
          Daniel Dai added a comment - PIG-2125 -5.patch committed to both trunk and 0.9 branch.
          Hide
          Daniel Dai added a comment -

          Steps to build pig.jar for hadoop 23:
          1. change "build.xml", modify "hadoopversion" to "23"
          2. change "ivy.xml", remove the lines:
          <dependency org="org.apache.hadoop" name="hadoop-core" rev="$

          {hadoop-core.version}

          "
          conf="compile->default;test->default"/>
          <dependency org="org.apache.hadoop" name="hadoop-test" rev="$

          {hadoop-test.version}

          "
          conf="compile->default;test->default"/>
          3. Build hadoop 23, copy hadoop 23 libraries into lib, here is a list:
          aopalliance-1.0.jar
          commons-cli-1.2.jar
          commons-codec-1.4.jar
          commons-collections-3.2.1.jar
          commons-configuration-1.6.jar
          commons-httpclient-3.1.jar
          guice-2.0.jar
          guice-servlet-2.0.jar
          hadoop-common-0.22.0-SNAPSHOT.jar
          hadoop-common-test-0.22.0-SNAPSHOT.jar
          hadoop-hdfs-0.22.0-SNAPSHOT.jar
          hadoop-hdfs-test-0.22.0-SNAPSHOT.jar
          hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar
          hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar
          hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar
          hadoop-mapreduce-client-hs-1.0-SNAPSHOT.jar
          hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
          hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT-tests.jar
          hadoop-mapreduce-client-shuffle-1.0-SNAPSHOT.jar
          hadoop-metrics2-0.22.0-SNAPSHOT.jar
          jetty-6.1.26.jar
          log4j-1.2.16.jar
          protobuf-java-2.4.0a.jar
          servlet-api-2.5-20081211.jar
          slf4j-api-1.6.1.jar
          slf4j-log4j12-1.6.1.jar
          yarn-api-1.0-SNAPSHOT.jar
          yarn-common-1.0-SNAPSHOT.jar
          yarn-server-common-1.0-SNAPSHOT.jar
          yarn-server-nodemanager-1.0-SNAPSHOT.jar
          yarn-server-resourcemanager-1.0-SNAPSHOT.jar
          yarn-server-tests-1.0-SNAPSHOT-tests.jar

          4. Then, do an "ant clean jar"
          5. You may also need to recompile your udf with the new hadoop libraries.

          Show
          Daniel Dai added a comment - Steps to build pig.jar for hadoop 23: 1. change "build.xml", modify "hadoopversion" to "23" 2. change "ivy.xml", remove the lines: <dependency org="org.apache.hadoop" name="hadoop-core" rev="$ {hadoop-core.version} " conf="compile->default;test->default"/> <dependency org="org.apache.hadoop" name="hadoop-test" rev="$ {hadoop-test.version} " conf="compile->default;test->default"/> 3. Build hadoop 23, copy hadoop 23 libraries into lib, here is a list: aopalliance-1.0.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.1.jar commons-configuration-1.6.jar commons-httpclient-3.1.jar guice-2.0.jar guice-servlet-2.0.jar hadoop-common-0.22.0-SNAPSHOT.jar hadoop-common-test-0.22.0-SNAPSHOT.jar hadoop-hdfs-0.22.0-SNAPSHOT.jar hadoop-hdfs-test-0.22.0-SNAPSHOT.jar hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar hadoop-mapreduce-client-hs-1.0-SNAPSHOT.jar hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT-tests.jar hadoop-mapreduce-client-shuffle-1.0-SNAPSHOT.jar hadoop-metrics2-0.22.0-SNAPSHOT.jar jetty-6.1.26.jar log4j-1.2.16.jar protobuf-java-2.4.0a.jar servlet-api-2.5-20081211.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar yarn-api-1.0-SNAPSHOT.jar yarn-common-1.0-SNAPSHOT.jar yarn-server-common-1.0-SNAPSHOT.jar yarn-server-nodemanager-1.0-SNAPSHOT.jar yarn-server-resourcemanager-1.0-SNAPSHOT.jar yarn-server-tests-1.0-SNAPSHOT-tests.jar 4. Then, do an "ant clean jar" 5. You may also need to recompile your udf with the new hadoop libraries.
          Hide
          Daniel Dai added a comment -

          Attach ContextFactory.java which eventually will be in hadoop's codebase. But for now, we need to put it in Pig's source tree in order to compile.

          Show
          Daniel Dai added a comment - Attach ContextFactory.java which eventually will be in hadoop's codebase. But for now, we need to put it in Pig's source tree in order to compile.
          Daniel Dai made changes -
          Attachment ContextFactory.java [ 12489093 ]
          Hide
          Daniel Dai added a comment -

          Put ContextFactory.java into $PIG_HOME/src/org/apache/hadoop/mapreduce, then compile.

          Show
          Daniel Dai added a comment - Put ContextFactory.java into $PIG_HOME/src/org/apache/hadoop/mapreduce, then compile.
          Hide
          Daniel Dai added a comment -

          PIG-2125-6.patch fixed many of core unit test failures for hadoop 23. There are still 20 failures left. I am working with Hadoop 23 team on these issues:
          I am running Pig unit tests with hadoop 23. Here is some issues I current find:

          1. Stability issue in MiniMRCluster. Some are intermittent, some fails consistently. Here is the stack I saw:
          java.io.FileNotFoundException:
          file:/home/daniel/pig/target/PigMiniCluster/PigMiniCluster-localDir/usercache/daniel/appcache/application_1313021517794_0001/container_1313021517794_0002_000001.tokens
          at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:187)
          at org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150)
          at org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595)
          at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:188)
          at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:736)
          at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:733)
          at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2278)
          at org.apache.hadoop.fs.FileContext.open(FileContext.java:733)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:137)
          at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:85)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:779)

          If success, I didn't see it in the log

          2. Couter issue. Seems most counter tests fail.
          JobClient.getJob(job.getAssignedJobID()) always give me null.

          3. MiniMRYarnCluster.getConfig() give me this entry:
          "mapreduce.job.hdfs-servers=$

          {fs.default.name}

          ", which should have done the variable substitution.

          4. Failure to start MiniHBaseCluster, all HBase tests fail.

          Show
          Daniel Dai added a comment - PIG-2125 -6.patch fixed many of core unit test failures for hadoop 23. There are still 20 failures left. I am working with Hadoop 23 team on these issues: I am running Pig unit tests with hadoop 23. Here is some issues I current find: 1. Stability issue in MiniMRCluster. Some are intermittent, some fails consistently. Here is the stack I saw: java.io.FileNotFoundException: file:/home/daniel/pig/target/PigMiniCluster/PigMiniCluster-localDir/usercache/daniel/appcache/application_1313021517794_0001/container_1313021517794_0002_000001.tokens at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:187) at org.apache.hadoop.fs.DelegateToFileSystem.open(DelegateToFileSystem.java:150) at org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:595) at org.apache.hadoop.fs.FilterFs.open(FilterFs.java:188) at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:736) at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:733) at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2278) at org.apache.hadoop.fs.FileContext.open(FileContext.java:733) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:137) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:85) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:779) If success, I didn't see it in the log 2. Couter issue. Seems most counter tests fail. JobClient.getJob(job.getAssignedJobID()) always give me null. 3. MiniMRYarnCluster.getConfig() give me this entry: "mapreduce.job.hdfs-servers=$ {fs.default.name} ", which should have done the variable substitution. 4. Failure to start MiniHBaseCluster, all HBase tests fail.
          Daniel Dai made changes -
          Attachment PIG-2125-6.patch [ 12490706 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to MAPREDUCE-279 [ MAPREDUCE-279 ]
          Hide
          Daniel Dai added a comment -

          PIG-2125-7.patch catch up with Pig trunk, and compilable against Hadoop 0.23 brach.

          Show
          Daniel Dai added a comment - PIG-2125 -7.patch catch up with Pig trunk, and compilable against Hadoop 0.23 brach.
          Daniel Dai made changes -
          Attachment PIG-2125-7.patch [ 12496952 ]
          Hide
          Daniel Dai added a comment -

          e2e-hadoop23.patch is a separable patch from PIG-2125-7.patch which enable e2e test only.

          Show
          Daniel Dai added a comment - e2e-hadoop23.patch is a separable patch from PIG-2125 -7.patch which enable e2e test only.
          Daniel Dai made changes -
          Attachment e2e-hadoop23.patch [ 12497067 ]
          Daniel Dai made changes -
          Attachment PIG-2125-7.patch [ 12496952 ]
          Hide
          Daniel Dai added a comment -

          Missing one file in PIG-2125-7.patch, reattach.

          Show
          Daniel Dai added a comment - Missing one file in PIG-2125 -7.patch, reattach.
          Daniel Dai made changes -
          Attachment PIG-2125-7.patch [ 12497099 ]
          Hide
          Daniel Dai added a comment -

          The same patch for 0.9 branch.

          Show
          Daniel Dai added a comment - The same patch for 0.9 branch.
          Daniel Dai made changes -
          Attachment PIG-2125-7_0.9.patch [ 12497103 ]
          Arun C Murthy made changes -
          Link This issue is blocked by MAPREDUCE-3138 [ MAPREDUCE-3138 ]
          Hide
          Arun C Murthy added a comment -

          I'll get ContextFactory committed to branch-0.23 so that Pig doesn't need to own this.

          Show
          Arun C Murthy added a comment - I'll get ContextFactory committed to branch-0.23 so that Pig doesn't need to own this.
          Hide
          Arun C Murthy added a comment -

          I've committed MAPREDUCE-3138 i.e. the ContextFactory patch to Hadoop MapReduce. Thanks.

          Show
          Arun C Murthy added a comment - I've committed MAPREDUCE-3138 i.e. the ContextFactory patch to Hadoop MapReduce. Thanks.
          Hide
          Thomas Weise added a comment -

          This patch is to support hadoopversion=23 to resolve dependencies through maven.

          Note that currently mapreduce does not have the jar files in maven, so you will still need to copy those into lib/

          Show
          Thomas Weise added a comment - This patch is to support hadoopversion=23 to resolve dependencies through maven. Note that currently mapreduce does not have the jar files in maven, so you will still need to copy those into lib/
          Thomas Weise made changes -
          Attachment PIG-2125-ivy-0.9.patch [ 12497813 ]
          Hide
          Thomas Weise added a comment -

          PIG-2125-buildxml-0.9.patch - Changes to build.xml for buildJar target. Verified changes for .20 build via diff of jar tv ... before and after.

          File list for .23 fat jar needs to be tuned once all maven dependencies are in place and tests compile/pass.

          Show
          Thomas Weise added a comment - PIG-2125 -buildxml-0.9.patch - Changes to build.xml for buildJar target. Verified changes for .20 build via diff of jar tv ... before and after. File list for .23 fat jar needs to be tuned once all maven dependencies are in place and tests compile/pass.
          Thomas Weise made changes -
          Attachment PIG-2125-buildxml-0.9.patch [ 12497937 ]
          Hide
          Daniel Dai added a comment -

          PIG-2125-commitJob.patch is to address incompatibility change introduced in MAPREDUCE-2702

          Show
          Daniel Dai added a comment - PIG-2125 -commitJob.patch is to address incompatibility change introduced in MAPREDUCE-2702
          Daniel Dai made changes -
          Attachment PIG-2125-commitJob.patch [ 12498820 ]
          Daniel Dai made changes -
          Attachment PIG-2125-commitJob.patch [ 12498820 ]
          Daniel Dai made changes -
          Attachment PIG-2125-commitJob.patch [ 12498903 ]
          Hide
          Daniel Dai added a comment -

          Tests pass for PIG-2125-commitJob.patch. No test included since it solve the problem only in hadoop 23, which is not released.

          Show
          Daniel Dai added a comment - Tests pass for PIG-2125 -commitJob.patch. No test included since it solve the problem only in hadoop 23, which is not released.
          Hide
          Olga Natkovich added a comment -

          Did the tests pass for both .Next and 20.x version?

          Show
          Olga Natkovich added a comment - Did the tests pass for both .Next and 20.x version?
          Hide
          Daniel Dai added a comment -

          All tests done on 20.X. I only did sanity tests on .Next, since I don't have the facility to run the whole thing currently.

          Show
          Daniel Dai added a comment - All tests done on 20.X. I only did sanity tests on .Next, since I don't have the facility to run the whole thing currently.
          Hide
          Olga Natkovich added a comment -

          I think that is good enough. As long as we do not regress on the current Hadoop I think we are good. Just need a review then.

          Show
          Olga Natkovich added a comment - I think that is good enough. As long as we do not regress on the current Hadoop I think we are good. Just need a review then.
          Hide
          Thejas M Nair added a comment -

          +1 for PIG-2125-commitJob.patch

          Show
          Thejas M Nair added a comment - +1 for PIG-2125 -commitJob.patch
          Hide
          Daniel Dai added a comment -

          PIG-2125-zebra.patch fix zebra test failures introduced by PIG-2125-commitJob.patch

          Show
          Daniel Dai added a comment - PIG-2125 -zebra.patch fix zebra test failures introduced by PIG-2125 -commitJob.patch
          Daniel Dai made changes -
          Attachment PIG-2125-zebra.patch [ 12499231 ]
          Hide
          Thomas Weise added a comment -

          Update to the mapreduce dependencies (replaces previous patch).

          Show
          Thomas Weise added a comment - Update to the mapreduce dependencies (replaces previous patch).
          Thomas Weise made changes -
          Attachment PIG-2125-ivy-0.9-2.patch [ 12499625 ]
          Hide
          Thomas Weise added a comment -

          Modified ivy patch to pull compile-test dependencies correctly. Currently the snapshots are outdated, compile-test should pass once they are refreshed.

          Show
          Thomas Weise added a comment - Modified ivy patch to pull compile-test dependencies correctly. Currently the snapshots are outdated, compile-test should pass once they are refreshed.
          Thomas Weise made changes -
          Attachment PIG-2125-ivy-0.9-3.patch [ 12499724 ]
          Thomas Weise made changes -
          Attachment PIG-2125-ivy-0.9-2.patch [ 12499625 ]
          Thomas Weise made changes -
          Attachment PIG-2125-ivy-0.9.patch [ 12497813 ]
          Hide
          Daniel Dai added a comment -

          PIG-2125-8.patch makes Pig compile and run against both 23 and 20 using maven artifacts. The only exception is hadoop-mapreduce.jar, which hadoop still fail to publish, and we need to copy it into lib. PIG-2125-8.patch include all changes in PIG-2125-7.patch, PIG-2125-buildxml-0.9.patch and PIG-2125-ivy-0.9-3.patch.

          Show
          Daniel Dai added a comment - PIG-2125 -8.patch makes Pig compile and run against both 23 and 20 using maven artifacts. The only exception is hadoop-mapreduce.jar, which hadoop still fail to publish, and we need to copy it into lib. PIG-2125 -8.patch include all changes in PIG-2125 -7.patch, PIG-2125 -buildxml-0.9.patch and PIG-2125 -ivy-0.9-3.patch.
          Daniel Dai made changes -
          Attachment PIG-2125-8.patch [ 12501721 ]
          Hide
          Daniel Dai added a comment -

          PIG-2125-9.patch & PIG-2125-9_0.9.patch compile and run Pig without any manual intervention.

          Show
          Daniel Dai added a comment - PIG-2125 -9.patch & PIG-2125 -9_0.9.patch compile and run Pig without any manual intervention.
          Daniel Dai made changes -
          Attachment PIG-2125-9.patch [ 12501872 ]
          Attachment PIG-2125-9_0.9.patch [ 12501873 ]
          Daniel Dai made changes -
          Attachment PIG-2125-9_0.9.patch [ 12501873 ]
          Daniel Dai made changes -
          Attachment PIG-2125-9_0.9.patch [ 12501876 ]
          Hide
          Thomas Weise added a comment -

          +1 ant compile-test is good on branch-0.9 now

          Show
          Thomas Weise added a comment - +1 ant compile-test is good on branch-0.9 now
          Hide
          Daniel Dai added a comment -

          PIG-2125-10.patch, PIG-2125-10_0.9.patch fix test-patch warnings:
          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 46 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 release audit. The applied patch generated 462 release audit warnings (more than the trunk's current 447 warnings).

          All new files has proper header.

          Unit test pass.

          Committed to trunk, 0.10 branch and 0.9 branch.

          Show
          Daniel Dai added a comment - PIG-2125 -10.patch, PIG-2125 -10_0.9.patch fix test-patch warnings: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 46 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 462 release audit warnings (more than the trunk's current 447 warnings). All new files has proper header. Unit test pass. Committed to trunk, 0.10 branch and 0.9 branch.
          Daniel Dai made changes -
          Attachment PIG-2125-10.patch [ 12502024 ]
          Attachment PIG-2125-10_0.9.patch [ 12502025 ]
          Hide
          Daniel Dai added a comment -

          After PIG-2125-10.patch, we can compile and run end-to-end tests on 23. There are still ~20 unit tests fail on 23 and we are working on that.

          Since all patches attached to the Jira is committed, and this thread is becoming big and confusing, I would suggest to close this ticket and create a separate ticket to track unit test fixes for hadoop 23, agree?

          Show
          Daniel Dai added a comment - After PIG-2125 -10.patch, we can compile and run end-to-end tests on 23. There are still ~20 unit tests fail on 23 and we are working on that. Since all patches attached to the Jira is committed, and this thread is becoming big and confusing, I would suggest to close this ticket and create a separate ticket to track unit test fixes for hadoop 23, agree?
          Daniel Dai made changes -
          Fix Version/s 0.9.2 [ 12318248 ]
          Affects Version/s 0.9.1 [ 12317343 ]
          Daniel Dai made changes -
          Link This issue incorporates PIG-2347 [ PIG-2347 ]
          Hide
          Daniel Dai added a comment -

          Move the remaining work to PIG-2347.

          Show
          Daniel Dai added a comment - Move the remaining work to PIG-2347 .
          Daniel Dai made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Thomas Weise made changes -
          Link This issue relates to HIVE-2468 [ HIVE-2468 ]
          Thomas Weise made changes -
          Link This issue relates to PIG-2277 [ PIG-2277 ]
          Thomas Weise made changes -
          Link This issue relates to PIG-2388 [ PIG-2388 ]
          Thomas Weise made changes -
          Link This issue blocks HCATALOG-179 [ HCATALOG-179 ]
          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue blocks HCATALOG-179 [ HCATALOG-179 ]
          Gavin made changes -
          Link This issue is depended upon by HCATALOG-179 [ HCATALOG-179 ]

            People

            • Assignee:
              Daniel Dai
              Reporter:
              Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development