Uploaded image for project: 'MRUnit'
  1. MRUnit
  2. MRUNIT-56

0.8.0 release does not work with Hadoop 0.23

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.1
    • Labels:
      None

      Description

      Unfortunately MRUNIT-31doesn't fix this. I get this failure:

      java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskInputOutputContext, but class was expected
              at org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper.createCommon(MockContextWrapper.java:51)
              at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.create(MockMapContextWrapper.java:65)
              at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.<init>(MockMapContextWrapper.java:57)
              at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:195)
              at org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:185)
              at v5.MaxTemperatureMapperTest.parsesMissingTemperature(MaxTemperatureMapperTest.java:34)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      

        Activity

        Hide
        tomwhite Tom White added a comment -

        The problem is that the context objects changed from abstract classes to interfaces, and method invocation uses different bytecodes for the two, so the approach in MockContextWrapper won't work.

        It might be possible to use (more) reflection to avoid this, but it might be simpler to publish two artifacts.

        Show
        tomwhite Tom White added a comment - The problem is that the context objects changed from abstract classes to interfaces, and method invocation uses different bytecodes for the two, so the approach in MockContextWrapper won't work. It might be possible to use (more) reflection to avoid this, but it might be simpler to publish two artifacts.
        Hide
        brocknoland Brock Noland added a comment -

        I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against.

        From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20?

        Show
        brocknoland Brock Noland added a comment - I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against. From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20?
        Hide
        brocknoland Brock Noland added a comment -

        Via profiles I was able to get a tar.gz file with a version of the code built with and 0.20 and 0.23 branch:

        [noland@localhost mrunit-0.8.0-incubating]$ tar -ztvf target/mrunit-0.8.0-incubating-dist.tar.gz  | grep jar
        -rw-rw-r-- noland/noland 77811 2012-02-04 20:09 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-0.23.1-SNAPSHOT.jar
        -rw-rw-r-- noland/noland 75681 2012-02-04 20:10 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-hadoop-0.20.jar
        

        I used profiles as specified here:

        http://stackoverflow.com/questions/2132958/ant-to-maven-multiple-build-targets/2140819#2140819

        along with the current profiles we had. However, I am not clear on how this will work when we deploy to the maven repo since it appears to do a clean and I had to do mvn package -Phadoop.20 and separately mvn package -Phadoop-0.23 to build the two jars.

        mvn clean deploy -Pdeploy -Pjavadoc
        

        Next week I should have my maven book back which will probably help.

        Show
        brocknoland Brock Noland added a comment - Via profiles I was able to get a tar.gz file with a version of the code built with and 0.20 and 0.23 branch: [noland@localhost mrunit-0.8.0-incubating]$ tar -ztvf target/mrunit-0.8.0-incubating-dist.tar.gz | grep jar -rw-rw-r-- noland/noland 77811 2012-02-04 20:09 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-0.23.1-SNAPSHOT.jar -rw-rw-r-- noland/noland 75681 2012-02-04 20:10 mrunit-0.8.0-incubating/mrunit-0.8.0-incubating-hadoop-0.20.jar I used profiles as specified here: http://stackoverflow.com/questions/2132958/ant-to-maven-multiple-build-targets/2140819#2140819 along with the current profiles we had. However, I am not clear on how this will work when we deploy to the maven repo since it appears to do a clean and I had to do mvn package -Phadoop.20 and separately mvn package -Phadoop-0.23 to build the two jars. mvn clean deploy -Pdeploy -Pjavadoc Next week I should have my maven book back which will probably help.
        Hide
        tomwhite Tom White added a comment -

        > I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against.

        That's correct.

        > From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20?

        That's probably the best solution. How about mrunit-0.8.0 and mrunit-hadoop1-0.8.0 for the names?

        Show
        tomwhite Tom White added a comment - > I could be wrong, but the code itself works with both versions but the jar we distribute only works with the version we compiled against. That's correct. > From a tarball perspective, we could provide a single tar with two jars inside. WIth maven, I guess users could select mrunit-0.8.0 or mrunit-0.8.0-hadoop-0.20? That's probably the best solution. How about mrunit-0.8.0 and mrunit-hadoop1-0.8.0 for the names?
        Hide
        tomwhite Tom White added a comment -

        Also, it would be good to create a test module to run an integration test against these MRUnit JAR files.

        Show
        tomwhite Tom White added a comment - Also, it would be good to create a test module to run an integration test against these MRUnit JAR files.
        Hide
        brocknoland Brock Noland added a comment -

        Attached allows you to build a hadoop 1.0 version of the jar and a 0.23 version. This is not intended for commit.

        Show
        brocknoland Brock Noland added a comment - Attached allows you to build a hadoop 1.0 version of the jar and a 0.23 version. This is not intended for commit.
        Hide
        brocknoland Brock Noland added a comment -

        Using profiles (attached patch) it looks like it's possible to build the two jars needed with two separate invocations of the mvn package. The steps are as follows:

        $ mvn clean package -Phadoop-1.0 -Dhadoop.version=1.0.0
        $ mv target/mrunit-0.8.0-incubating-hadoop-1.0.jar /tmp/
        $ mvn clean assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT
        $ mv /tmp/mrunit-0.8.0-incubating-hadoop-1.0.jar target/
        $ mvn assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT
        $ mvn deploy -Pdeploy -Pjavadoc -Dhadoop.version=0.23.1-SNAPSHOT

        This gives you a tar which contain two builds of mrunit:

        mrunit-0.8.0-incubating-hadoop-1.0.jar (for 1.0)
        mrunit-0.8.0-incubating.jar (for 0.23.1)

        The maven repo contains the archive but not the two jar files directly. It only contains mrunit-0.8.0-incubating.jar

        https://repository.apache.org/content/repositories/orgapachemrunit-205/

        I think this gives us what we want but I am guessing there is a much better way of doing this.

        Show
        brocknoland Brock Noland added a comment - Using profiles (attached patch) it looks like it's possible to build the two jars needed with two separate invocations of the mvn package. The steps are as follows: $ mvn clean package -Phadoop-1.0 -Dhadoop.version=1.0.0 $ mv target/mrunit-0.8.0-incubating-hadoop-1.0.jar /tmp/ $ mvn clean assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT $ mv /tmp/mrunit-0.8.0-incubating-hadoop-1.0.jar target/ $ mvn assembly:assembly verify -Phadoop-0.23 -Dhadoop.version=0.23.1-SNAPSHOT $ mvn deploy -Pdeploy -Pjavadoc -Dhadoop.version=0.23.1-SNAPSHOT This gives you a tar which contain two builds of mrunit: mrunit-0.8.0-incubating-hadoop-1.0.jar (for 1.0) mrunit-0.8.0-incubating.jar (for 0.23.1) The maven repo contains the archive but not the two jar files directly. It only contains mrunit-0.8.0-incubating.jar https://repository.apache.org/content/repositories/orgapachemrunit-205/ I think this gives us what we want but I am guessing there is a much better way of doing this.
        Hide
        brocknoland Brock Noland added a comment -

        > Also, it would be good to create a test module to run an integration test against these MRUnit JAR files.

        There is this:

        src/test/sh/run-artifact-unit-tests.sh

        but it should be modified to take the mrunit jar and hadoop jars on the command line as opposed to assuming them.

        Show
        brocknoland Brock Noland added a comment - > Also, it would be good to create a test module to run an integration test against these MRUnit JAR files. There is this: src/test/sh/run-artifact-unit-tests.sh but it should be modified to take the mrunit jar and hadoop jars on the command line as opposed to assuming them.
        Hide
        tucu00 Alejandro Abdelnur added a comment -

        I'd suggest we publish 2 different versions for the same groupId/artifact, with the version of hadoop encoded in the version. By doing this users will get Maven to do dependency version resolution, otherwise for Maven the JARs would be for different entities and you could potentially end up with the 2 JARs in your project.

        Show
        tucu00 Alejandro Abdelnur added a comment - I'd suggest we publish 2 different versions for the same groupId/artifact, with the version of hadoop encoded in the version. By doing this users will get Maven to do dependency version resolution, otherwise for Maven the JARs would be for different entities and you could potentially end up with the 2 JARs in your project.
        Hide
        brocknoland Brock Noland added a comment -

        OK, so we could change our release process to do the following:

        Create a branch mrunit-X.Y.Z
        Copy the branch to mrunit-hadoop-1.0-X.Y.Z
        Change the pom version to mrunit-hadoop-1.0-X.Y.Z
        Compile mrunit-X.Y.Z against 0.23 or newer
        Compile mrunit-hadoop-1.0-X.Y.Z against hadoop 1.X or newer
        Package and deploy to maven both versions

        Thoughts? Easier method?

        Show
        brocknoland Brock Noland added a comment - OK, so we could change our release process to do the following: Create a branch mrunit-X.Y.Z Copy the branch to mrunit-hadoop-1.0-X.Y.Z Change the pom version to mrunit-hadoop-1.0-X.Y.Z Compile mrunit-X.Y.Z against 0.23 or newer Compile mrunit-hadoop-1.0-X.Y.Z against hadoop 1.X or newer Package and deploy to maven both versions Thoughts? Easier method?
        Hide
        brocknoland Brock Noland added a comment -

        1244827 in 0.8 and 1244828 in trunk

        Show
        brocknoland Brock Noland added a comment - 1244827 in 0.8 and 1244828 in trunk
        Hide
        jdonofrio Jim Donofrio added a comment -

        Shouldnt the version for 0.23 be changed from <specificHadoopVersion>0.23.1-SNAPSHOT</specificHadoopVersion> to <specificHadoopVersion>0.23.1</specificHadoopVersion>

        Show
        jdonofrio Jim Donofrio added a comment - Shouldnt the version for 0.23 be changed from <specificHadoopVersion>0.23.1-SNAPSHOT</specificHadoopVersion> to <specificHadoopVersion>0.23.1</specificHadoopVersion>
        Hide
        brocknoland Brock Noland added a comment -

        Yes, now that it has released we should update the version...

        Show
        brocknoland Brock Noland added a comment - Yes, now that it has released we should update the version...

          People

          • Assignee:
            brocknoland Brock Noland
            Reporter:
            tomwhite Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development