Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2002

MRUnit driver classes should provide ability to set a configuration object to be passed into the mapper/reducer



    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 0.20.2
    • None
    • contrib/mrunit
    • None


      Short description:

      Enhance the org.apache.hadoop.mrunit.mapreduce.MapDriver, ReduceDriver, and MapReduceDriver unit test driver classes to contain "setConfiguration" and "withConfiguration" methods for passing in user-supplied org.apache.hadoop.conf.Configuration objects, and have those configuration objects eventually get passed on to the Context objects that are passed in to the mapper/reducer "setup" methods. (Rather than passing in an empty Configuration object, as is being done now.)

      Long description:

      The MRUnit driver classes (i.e., MapDriver, ReduceDriver, and MapReduceDriver) ought to be enhanced to contain methods for setting a Configuration object to be used by the mapper/reducer being tested - i.e., setConfiguration() and withConfiguration().

      The only way to effectively pass parameters into a mapper or reducer is by setting properties on a configuration object, which the mapper/reducer can then retrieve in their "setup" step, and use to customize its operation. As a result, specific mappers/reducers may require the presence of specific configuration properties/parameters in order to function correctly (or at all). (I am currently coding such a reducer right now.)

      Testing such a mapper/reducer thus requires that the unit testing framework used provide the ability to pass in user-supplied Configuration objects to them so that they can be tested with appropriate parameter values. However, MRUnit currently does not provide this ability. (All mappers/reducers are always passed an empty configuration object.) And there is not even currently any (easy) way for the end-user to fix this problem by creating a simple sub-class that supplies this functionality, as such subclasses would require a substantial reimplementation/override of several MRUnit framework classes.

      I believe this something that is not too difficult to fix in the MRUnit framework code, however, and would greatly help the usability of MRUnit.

      Although I don't have time to code this enhancement right now, if needed/preferred I could squeeze out some time to code up a patch for this. If that's needed, please let me know.


        Issue Links



              Unassigned Unassigned
              darose David Rosenstrauch
              0 Vote for this issue
              0 Start watching this issue