MRUnit
  1. MRUnit
  2. MRUNIT-101

allow use of real OutputFormats such as TextOutputFormat instead of the MockOutputCollector/MockContextWrapper classes

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.0.0
    • Labels:

      Description

      Currently we only use the MockOutputCollector/MockContextWrapper which just copy the output key, value into a list. This prevents users from testing the outputformat for classes such as TextOutputFormat which should compare the toString versions of objects for equality not the objects themselves.

        Activity

        Hide
        Jim Donofrio added a comment -

        I will add a method withOutputFormat(Class<? extends FileOutputFormat> outputFormat, Class<? extends FileInputFormat> inputFormat). I think we should move to users passing in classes to be more consistent with the conf instead of instances of outputformats, mappers, etc. We can only really test classes that extend FileOutputFormat because there is no contract on how to add input/output to the other types. I will create a random dir mrunit-random number in java.io.tmpdir. I will add this path to the FileOutputFormat. I will use the FileOutputFormat to get a record writer and write out the output using a MockOutputCollector. Then using the provided FileInputFormat I will add the output file and call getSplits and then getRecordReader which will allow me to read the output back in and compare the results to the expected output using the traditional validate method

        Show
        Jim Donofrio added a comment - I will add a method withOutputFormat(Class<? extends FileOutputFormat> outputFormat, Class<? extends FileInputFormat> inputFormat). I think we should move to users passing in classes to be more consistent with the conf instead of instances of outputformats, mappers, etc. We can only really test classes that extend FileOutputFormat because there is no contract on how to add input/output to the other types. I will create a random dir mrunit-random number in java.io.tmpdir. I will add this path to the FileOutputFormat. I will use the FileOutputFormat to get a record writer and write out the output using a MockOutputCollector. Then using the provided FileInputFormat I will add the output file and call getSplits and then getRecordReader which will allow me to read the output back in and compare the results to the expected output using the traditional validate method
        Hide
        Jim Donofrio added a comment -

        initial checkin in 1336519, still need to cleanup the javadoc

        Show
        Jim Donofrio added a comment - initial checkin in 1336519, still need to cleanup the javadoc
        Hide
        Hudson added a comment -

        Integrated in mrunit-trunk #252 (See https://builds.apache.org/job/mrunit-trunk/252/)
        MRUNIT-101: allow use of real OutputFormats such as TextOutputFormat (Revision 6a5e5fbae2f11d6b8088a9c09a47056c49a4557f)
        MRUNIT-101: allow use of real OutputFormats such as TextOutputFormat (Revision 3b1f193d77590150e117d69f7a32c2c6e6c72051)

        Result = SUCCESS
        jdonofrio : https://git-wip-us.apache.org/repos/asf?p=mrunit.git&a=commit&h=6a5e5fbae2f11d6b8088a9c09a47056c49a4557f
        Files :

        • src/main/java/org/apache/hadoop/mrunit/ReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/MapDriver.java
        • src/main/java/org/apache/hadoop/mrunit/mapreduce/ReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java

        jdonofrio : https://git-wip-us.apache.org/repos/asf?p=mrunit.git&a=commit&h=3b1f193d77590150e117d69f7a32c2c6e6c72051
        Files :

        • src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/mapreduce/ReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/MapDriver.java
        • src/main/java/org/apache/hadoop/mrunit/ReduceDriver.java
        • src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java
        Show
        Hudson added a comment - Integrated in mrunit-trunk #252 (See https://builds.apache.org/job/mrunit-trunk/252/ ) MRUNIT-101 : allow use of real OutputFormats such as TextOutputFormat (Revision 6a5e5fbae2f11d6b8088a9c09a47056c49a4557f) MRUNIT-101 : allow use of real OutputFormats such as TextOutputFormat (Revision 3b1f193d77590150e117d69f7a32c2c6e6c72051) Result = SUCCESS jdonofrio : https://git-wip-us.apache.org/repos/asf?p=mrunit.git&a=commit&h=6a5e5fbae2f11d6b8088a9c09a47056c49a4557f Files : src/main/java/org/apache/hadoop/mrunit/ReduceDriver.java src/main/java/org/apache/hadoop/mrunit/MapDriver.java src/main/java/org/apache/hadoop/mrunit/mapreduce/ReduceDriver.java src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java jdonofrio : https://git-wip-us.apache.org/repos/asf?p=mrunit.git&a=commit&h=3b1f193d77590150e117d69f7a32c2c6e6c72051 Files : src/main/java/org/apache/hadoop/mrunit/MapReduceDriver.java src/main/java/org/apache/hadoop/mrunit/mapreduce/MapReduceDriver.java src/main/java/org/apache/hadoop/mrunit/mapreduce/ReduceDriver.java src/main/java/org/apache/hadoop/mrunit/MapDriver.java src/main/java/org/apache/hadoop/mrunit/ReduceDriver.java src/main/java/org/apache/hadoop/mrunit/mapreduce/MapDriver.java

          People

          • Assignee:
            Jim Donofrio
            Reporter:
            Jim Donofrio
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development