MRUnit
  1. MRUnit
  2. MRUNIT-56

0.8.0 release does not work with Hadoop 0.23

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.1
    • Labels:
      None

      Description

      Unfortunately MRUNIT-31doesn't fix this. I get this failure:

      java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected
              at org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper.createCommon(MockContextWrapper.java:51)
              at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.create(MockMapContextWrapper.java:65)
              at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.<init>(MockMapContextWrapper.java:57)
              at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:195)
              at org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:185)
              at v5.MaxTemperatureMapperTest.parsesMissingTemperature(MaxTemperatureMapperTest.java:34)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      

        Activity

        Hide
        Tom White added a comment -

        The problem is that the context objects changed from abstract classes to interfaces, and method invocation uses different bytecodes for the two, so the approach in MockContextWrapper won't work.

        It might be possible to use (more) reflection to avoid this, but it might be simpler to publish two artifacts.

        Show
        Tom White added a comment - The problem is that the context objects changed from abstract classes to interfaces, and method invocation uses different bytecodes for the two, so the approach in MockContextWrapper won't work. It might be possible to use (more) reflection to avoid this, but it might be simpler to publish two artifacts.
        Hide
        Brock Noland added a comment -

        I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against.

        From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20?

        Show
        Brock Noland added a comment - I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against. From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20?
        Hide
        Brock Noland added a comment -

        Via profiles I was able to get a tar.gz file with a version of the code built with and 0.20 and 0.23 branch:

        [noland@localhost mrunit-0.8.0-incubating]$ tar -ztvf target/mrunit-0.8.0-incubating-dist.tar.gz  | grep jar
        -rw-rw-r-- noland/noland 77811 2012-02-04 20:09 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-0.23.1-SNAPSHOT.jar
        -rw-rw-r-- noland/noland 75681 2012-02-04 20:10 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-hadoop-0.20.jar
        

        I used profiles as specified here:

        http://stackoverflow.com/questions/2132958/ant-to-maven-multiple-build-targets/2140819#2140819

        along with the current profiles we had. However, I am not clear on how this will work when we deploy to the maven repo since it appears to do a clean and I had to do mvn package -Phadoop.20 and separately mvn package -Phadoop-0.23 to build the two jars.

        mvn clean deploy -Pdeploy -Pjavadoc
        

        Next week I should have my maven book back which will probably help.

        Show
        Brock Noland added a comment - Via profiles I was able to get a tar.gz file with a version of the code built with and 0.20 and 0.23 branch: [noland@localhost mrunit-0.8.0-incubating]$ tar -ztvf target/mrunit-0.8.0-incubating-dist.tar.gz | grep jar -rw-rw-r-- noland/noland 77811 2012-02-04 20:09 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-0.23.1-SNAPSHOT.jar -rw-rw-r-- noland/noland 75681 2012-02-04 20:10 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-hadoop-0.20.jar I used profiles as specified here: http://stackoverflow.com/questions/2132958/ant-to-maven-multiple-build-targets/2140819#2140819 along with the current profiles we had. However, I am not clear on how this will work when we deploy to the maven repo since it appears to do a clean and I had to do mvn package -Phadoop.20 and separately mvn package -Phadoop-0.23 to build the two jars. mvn clean deploy -Pdeploy -Pjavadoc Next week I should have my maven book back which will probably help.
        Hide
        Tom White added a comment -

        > I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against.

        That's correct.

        > From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20?

        That's probably the best solution. How about mrunit-0.8.0 and mrunit-hadoop1-0.8.0 for the names?

        Show
        Tom White added a comment - > I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against. That's correct. > From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20? That's probably the best solution. How about mrunit-0.8.0 and mrunit-hadoop1-0.8.0 for the names?
        Hide
        Tom White added a comment -

        Also, it would be good to create a test module to run an integration test against these MRUnit JAR files.

        Show
        Tom White added a comment - Also, it would be good to create a test module to run an integration test against these MRUnit JAR files.
        Hide
        Brock Noland added a comment -

        Attached allows you to build a hadoop 1.0 version of the jar and a 0.23 version. This is not intended for commit.

        Show
        Brock Noland added a comment - Attached allows you to build a hadoop 1.0 version of the jar and a 0.23 version. This is not intended for commit.
        Hide
        Brock Noland added a comment -

        Using profiles (attached patch) it looks like it's possible to build the two jars needed with two separate invocations of the mvn package. The steps are as follows:

        $ mvn clean package -Phadoop-1.0 -Dhadoop.version=1.0.0
        $ mv target/mrunit-0.8.0-incubating-hadoop-1.0.jar /tmp/
        $ mvn clean assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT
        $ mv /tmp/mrunit-0.8.0-incubating-hadoop-1.0.jar target/
        $ mvn assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT
        $ mvn deploy -Pdeploy -Pjavadoc -Dhadoop.version=0.23.1-SNAPSHOT

        This gives you a tar which contain two builds of mrunit:

        mrunit-0.8.0-incubating-hadoop-1.0.jar (for 1.0)
        mrunit-0.8.0-incubating.jar (for 0.23.1)

        The maven repo contains the archive but not the two jar files directly. It only contains mrunit-0.8.0-incubating.jar

        https://repository.apache.org/content/repositories/orgapachemrunit-205/

        I think this gives us what we want but I am guessing there is a much better way of doing this.

        Show
        Brock Noland added a comment - Using profiles (attached patch) it looks like it's possible to build the two jars needed with two separate invocations of the mvn package. The steps are as follows: $ mvn clean package -Phadoop-1.0 -Dhadoop.version=1.0.0 $ mv target/mrunit-0.8.0-incubating-hadoop-1.0.jar /tmp/ $ mvn clean assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT $ mv /tmp/mrunit-0.8.0-incubating-hadoop-1.0.jar target/ $ mvn assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT $ mvn deploy -Pdeploy -Pjavadoc -Dhadoop.version=0.23.1-SNAPSHOT This gives you a tar which contain two builds of mrunit: mrunit-0.8.0-incubating-hadoop-1.0.jar (for 1.0) mrunit-0.8.0-incubating.jar (for 0.23.1) The maven repo contains the archive but not the two jar files directly. It only contains mrunit-0.8.0-incubating.jar https://repository.apache.org/content/repositories/orgapachemrunit-205/ I think this gives us what we want but I am guessing there is a much better way of doing this.
        Hide
        Brock Noland added a comment -

        > Also, it would be good to create a test module to run an integration test against these MRUnit JAR files.

        There is this:

        src/test/sh/run-artifact-unit-tests.sh

        but it should be modified to take the mrunit jar and hadoop jars on the command line as opposed to assuming them.

        Show
        Brock Noland added a comment - > Also, it would be good to create a test module to run an integration test against these MRUnit JAR files. There is this: src/test/sh/run-artifact-unit-tests.sh but it should be modified to take the mrunit jar and hadoop jars on the command line as opposed to assuming them.
        Hide
        Alejandro Abdelnur added a comment -

        I'd suggest we publish 2 different versions for the same groupId/artifact, with the version of hadoop encoded in the version. By doing this users will get Maven to do dependency version resolution, otherwise for Maven the JARs would be for different entities and you could potentially end up with the 2 JARs in your project.

        Show
        Alejandro Abdelnur added a comment - I'd suggest we publish 2 different versions for the same groupId/artifact, with the version of hadoop encoded in the version. By doing this users will get Maven to do dependency version resolution, otherwise for Maven the JARs would be for different entities and you could potentially end up with the 2 JARs in your project.
        Hide
        Brock Noland added a comment -

        OK, so we could change our release process to do the following:

        Create a branch mrunit-X.Y.Z
        Copy the branch to mrunit-hadoop-1.0-X.Y.Z
        Change the pom version to mrunit-hadoop-1.0-X.Y.Z
        Compile mrunit-X.Y.Z against 0.23 or newer
        Compile mrunit-hadoop-1.0-X.Y.Z against hadoop 1.X or newer
        Package and deploy to maven both versions

        Thoughts? Easier method?

        Show
        Brock Noland added a comment - OK, so we could change our release process to do the following: Create a branch mrunit-X.Y.Z Copy the branch to mrunit-hadoop-1.0-X.Y.Z Change the pom version to mrunit-hadoop-1.0-X.Y.Z Compile mrunit-X.Y.Z against 0.23 or newer Compile mrunit-hadoop-1.0-X.Y.Z against hadoop 1.X or newer Package and deploy to maven both versions Thoughts? Easier method?
        Hide
        Brock Noland added a comment -

        1244827 in 0.8 and 1244828 in trunk

        Show
        Brock Noland added a comment - 1244827 in 0.8 and 1244828 in trunk
        Hide
        Jim Donofrio added a comment -

        Shouldnt the version for 0.23 be changed from <specificHadoopVersion>0.23.1-SNAPSHOT</specificHadoopVersion> to <specificHadoopVersion>0.23.1</specificHadoopVersion>

        Show
        Jim Donofrio added a comment - Shouldnt the version for 0.23 be changed from <specificHadoopVersion>0.23.1-SNAPSHOT</specificHadoopVersion> to <specificHadoopVersion>0.23.1</specificHadoopVersion>
        Hide
        Brock Noland added a comment -

        Yes, now that it has released we should update the version...

        Show
        Brock Noland added a comment - Yes, now that it has released we should update the version...

          People

          • Assignee:
            Brock Noland
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development