Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2546

build and publish Pig artifacts compiled against Hadoop 0.23

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 0.9.2
    • Fix Version/s: 0.11
    • Component/s: build
    • Labels:
      None

      Description

      The pig JAR available in Apache Maven repo has been compiled with a version of Hadoop prior to 0.23.

      This JAR cannot be used to run Pig against Hadoop 0.23 because of changes in the JobControl API [ie the JobControl.addJob(Job) method now is JobControl.addJob(ControlledJob), and while Job implements ControlledJob, because ControlledJob is an interface, the generated bytecode for the signature is different).

      I assume we'll need 2 different versions of the JARs, one for pre 0.23 and one for post 0.23.

      This affects Oozie integration with Hadoop 0.23

        Issue Links

          Activity

          Hide
          daijy Daniel Dai added a comment -

          Yes, we do need publish pig-0.9.2-23.jar. Some change in build.xml is needed.

          Show
          daijy Daniel Dai added a comment - Yes, we do need publish pig-0.9.2-23.jar. Some change in build.xml is needed.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          I was able to build/mvn-install pig using the -Dhadoopversion=23 flag and test it with Oozie/trunk and Hadoop/trunk.

          I guess the required changes in the build is to postfix the -23 to the version and to push that to the Apache Maven repo.

          And then, the jenkins job that pushes the JARs to the Apache Maven repo should run twice, once for Hadoop 1.x and once for Hadoop 0.23.

          Show
          tucu00 Alejandro Abdelnur added a comment - I was able to build/mvn-install pig using the -Dhadoopversion=23 flag and test it with Oozie/trunk and Hadoop/trunk. I guess the required changes in the build is to postfix the -23 to the version and to push that to the Apache Maven repo. And then, the jenkins job that pushes the JARs to the Apache Maven repo should run twice, once for Hadoop 1.x and once for Hadoop 0.23.
          Hide
          daijy Daniel Dai added a comment -

          Yes, we shall change build.xml to have a different name for 23 artifacts. That's it.

          Show
          daijy Daniel Dai added a comment - Yes, we shall change build.xml to have a different name for 23 artifacts. That's it.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          @daniel, as discussed in the alias. No different names, same groupId and same artifactId, just different classifier.

          Show
          tucu00 Alejandro Abdelnur added a comment - @daniel, as discussed in the alias. No different names, same groupId and same artifactId, just different classifier.
          Hide
          daijy Daniel Dai added a comment -

          Sounds good to me.

          Show
          daijy Daniel Dai added a comment - Sounds good to me.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          So, no chance to get this done for 0.9?

          Show
          tucu00 Alejandro Abdelnur added a comment - So, no chance to get this done for 0.9?
          Hide
          daijy Daniel Dai added a comment -

          Fix version was 0.9.2, since I close 0.9.2 tag, Jira move all 0.9.2 issues to 0.10. Feel free to correct it.

          Show
          daijy Daniel Dai added a comment - Fix version was 0.9.2, since I close 0.9.2 tag, Jira move all 0.9.2 issues to 0.10. Feel free to correct it.
          Hide
          daijy Daniel Dai added a comment -

          Talked with Giri, we might want to publish to a separate group pig23. Not only pig-core.jar changes, pig-test.jar also change for hadoop 23. So one single classifier might not solve the problem.

          Show
          daijy Daniel Dai added a comment - Talked with Giri, we might want to publish to a separate group pig23. Not only pig-core.jar changes, pig-test.jar also change for hadoop 23. So one single classifier might not solve the problem.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          is the pig-test.jar useful for downstream projects? If not publishing to maven is not required. then a classifier would be a option

          Show
          tucu00 Alejandro Abdelnur added a comment - is the pig-test.jar useful for downstream projects? If not publishing to maven is not required. then a classifier would be a option
          Hide
          daijy Daniel Dai added a comment -

          Might be, if they need to write unit test using Pig.

          Show
          daijy Daniel Dai added a comment - Might be, if they need to write unit test using Pig.
          Hide
          rohini Rohini Palaniswamy added a comment -

          Closing as duplicate of PIG-2907 as the patch is posted there even though this was created first and has more context.

          Show
          rohini Rohini Palaniswamy added a comment - Closing as duplicate of PIG-2907 as the patch is posted there even though this was created first and has more context.

            People

            • Assignee:
              rohini Rohini Palaniswamy
              Reporter:
              tucu00 Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development