Pig
  1. Pig
  2. PIG-2546

build and publish Pig artifacts compiled against Hadoop 0.23

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Duplicate
    • Affects Version/s: 0.9.2
    • Fix Version/s: 0.11
    • Component/s: build
    • Labels:
      None

      Description

      The pig JAR available in Apache Maven repo has been compiled with a version of Hadoop prior to 0.23.

      This JAR cannot be used to run Pig against Hadoop 0.23 because of changes in the JobControl API [ie the JobControl.addJob(Job) method now is JobControl.addJob(ControlledJob), and while Job implements ControlledJob, because ControlledJob is an interface, the generated bytecode for the signature is different).

      I assume we'll need 2 different versions of the JARs, one for pre 0.23 and one for post 0.23.

      This affects Oozie integration with Hadoop 0.23

        Issue Links

          Activity

          Hide
          Daniel Dai added a comment -

          Yes, we do need publish pig-0.9.2-23.jar. Some change in build.xml is needed.

          Show
          Daniel Dai added a comment - Yes, we do need publish pig-0.9.2-23.jar. Some change in build.xml is needed.
          Hide
          Alejandro Abdelnur added a comment -

          I was able to build/mvn-install pig using the -Dhadoopversion=23 flag and test it with Oozie/trunk and Hadoop/trunk.

          I guess the required changes in the build is to postfix the -23 to the version and to push that to the Apache Maven repo.

          And then, the jenkins job that pushes the JARs to the Apache Maven repo should run twice, once for Hadoop 1.x and once for Hadoop 0.23.

          Show
          Alejandro Abdelnur added a comment - I was able to build/mvn-install pig using the -Dhadoopversion=23 flag and test it with Oozie/trunk and Hadoop/trunk. I guess the required changes in the build is to postfix the -23 to the version and to push that to the Apache Maven repo. And then, the jenkins job that pushes the JARs to the Apache Maven repo should run twice, once for Hadoop 1.x and once for Hadoop 0.23.
          Hide
          Daniel Dai added a comment -

          Yes, we shall change build.xml to have a different name for 23 artifacts. That's it.

          Show
          Daniel Dai added a comment - Yes, we shall change build.xml to have a different name for 23 artifacts. That's it.
          Hide
          Alejandro Abdelnur added a comment -

          @daniel, as discussed in the alias. No different names, same groupId and same artifactId, just different classifier.

          Show
          Alejandro Abdelnur added a comment - @daniel, as discussed in the alias. No different names, same groupId and same artifactId, just different classifier.
          Hide
          Daniel Dai added a comment -

          Sounds good to me.

          Show
          Daniel Dai added a comment - Sounds good to me.
          Hide
          Alejandro Abdelnur added a comment -

          So, no chance to get this done for 0.9?

          Show
          Alejandro Abdelnur added a comment - So, no chance to get this done for 0.9?
          Hide
          Daniel Dai added a comment -

          Fix version was 0.9.2, since I close 0.9.2 tag, Jira move all 0.9.2 issues to 0.10. Feel free to correct it.

          Show
          Daniel Dai added a comment - Fix version was 0.9.2, since I close 0.9.2 tag, Jira move all 0.9.2 issues to 0.10. Feel free to correct it.
          Hide
          Daniel Dai added a comment -

          Talked with Giri, we might want to publish to a separate group pig23. Not only pig-core.jar changes, pig-test.jar also change for hadoop 23. So one single classifier might not solve the problem.

          Show
          Daniel Dai added a comment - Talked with Giri, we might want to publish to a separate group pig23. Not only pig-core.jar changes, pig-test.jar also change for hadoop 23. So one single classifier might not solve the problem.
          Hide
          Alejandro Abdelnur added a comment -

          is the pig-test.jar useful for downstream projects? If not publishing to maven is not required. then a classifier would be a option

          Show
          Alejandro Abdelnur added a comment - is the pig-test.jar useful for downstream projects? If not publishing to maven is not required. then a classifier would be a option
          Hide
          Daniel Dai added a comment -

          Might be, if they need to write unit test using Pig.

          Show
          Daniel Dai added a comment - Might be, if they need to write unit test using Pig.
          Hide
          Rohini Palaniswamy added a comment -

          Closing as duplicate of PIG-2907 as the patch is posted there even though this was created first and has more context.

          Show
          Rohini Palaniswamy added a comment - Closing as duplicate of PIG-2907 as the patch is posted there even though this was created first and has more context.

            People

            • Assignee:
              Rohini Palaniswamy
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development