MRUnit
  1. MRUnit
  2. MRUNIT-99

dont force K2 in MapReduceDriver's to be Comparable since Hadoop dont enforce this on the map output key and value classes

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.9.0
    • Labels:
      None

      Description

      We should not force K2 in MapReduceDriver's to be Comparable because Hadoop does not enforce this on the map output key and value classes. This makes MRUnit difficult to use when using non Writable classes that dont need to be Comparable but define their own key comparators in the JobConf

      I realized this after seeing [1] on stackoverflow. I am still trying to investigate if we have other conflicts with Avro

      [1]: http://stackoverflow.com/questions/10119616/use-mrunit-and-avro-together

        Activity

        Jim Donofrio created issue -
        Hide
        Jim Donofrio added a comment -

        With the release going out tomorrow I put together a quick patch. I removed extends Comparable from K2 in both MapReduceDriver classes. In MapReduceDriverBase if keyGroupComparator is null then I use new JobConf(getConfiguration()).getOutputValueGroupingComparator() which if the group comparator isnt set in the conf uses the key comparator and if that isnt set uses WritableComparator.get

        I think when designing the new api we can still have these convenience methods, withMapper, etc although we could get rid of them but we should use the conf to store all the settings to mimic hadoop. We could just have a getConf() method and allow users to set the necessary arguments in there. Then when we test the job we could throw a specific exception if they did not remember to set a property that we need in the conf

        committed in 1326461 in 0.9.0 and 1326463 in trunk

        Show
        Jim Donofrio added a comment - With the release going out tomorrow I put together a quick patch. I removed extends Comparable from K2 in both MapReduceDriver classes. In MapReduceDriverBase if keyGroupComparator is null then I use new JobConf(getConfiguration()).getOutputValueGroupingComparator() which if the group comparator isnt set in the conf uses the key comparator and if that isnt set uses WritableComparator.get I think when designing the new api we can still have these convenience methods, withMapper, etc although we could get rid of them but we should use the conf to store all the settings to mimic hadoop. We could just have a getConf() method and allow users to set the necessary arguments in there. Then when we test the job we could throw a specific exception if they did not remember to set a property that we need in the conf committed in 1326461 in 0.9.0 and 1326463 in trunk
        Jim Donofrio made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Jim Donofrio
            Reporter:
            Jim Donofrio
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development