[MAPREDUCE-2002] MRUnit driver classes should provide ability to set a configuration object to be passed into the mapper/reducer

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.20.2
• Component/s: contrib/mrunit
• Assignee: Unassigned
• Reporter: David Rosenstrauch
• Created: 11/Aug/10 02:43
• Updated: 15/Mar/12 07:47

Description

Short description:

Enhance the org.apache.hadoop.mrunit.mapreduce.MapDriver, ReduceDriver, and MapReduceDriver unit test driver classes to contain "setConfiguration" and "withConfiguration" methods for passing in user-supplied org.apache.hadoop.conf.Configuration objects, and have those configuration objects eventually get passed on to the Context objects that are passed in to the mapper/reducer "setup" methods. (Rather than passing in an empty Configuration object, as is being done now.)

Long description:

The MRUnit driver classes (i.e., MapDriver, ReduceDriver, and MapReduceDriver) ought to be enhanced to contain methods for setting a Configuration object to be used by the mapper/reducer being tested - i.e., setConfiguration() and withConfiguration().

The only way to effectively pass parameters into a mapper or reducer is by setting properties on a configuration object, which the mapper/reducer can then retrieve in their "setup" step, and use to customize its operation. As a result, specific mappers/reducers may require the presence of specific configuration properties/parameters in order to function correctly (or at all). (I am currently coding such a reducer right now.)

Testing such a mapper/reducer thus requires that the unit testing framework used provide the ability to pass in user-supplied Configuration objects to them so that they can be tested with appropriate parameter values. However, MRUnit currently does not provide this ability. (All mappers/reducers are always passed an empty configuration object.) And there is not even currently any (easy) way for the end-user to fix this problem by creating a simple sub-class that supplies this functionality, as such subclasses would require a substantial reimplementation/override of several MRUnit framework classes.

I believe this something that is not too difficult to fix in the MRUnit framework code, however, and would greatly help the usability of MRUnit.

Although I don't have time to code this enhancement right now, if needed/preferred I could squeeze out some time to code up a patch for this. If that's needed, please let me know.

Issue Links

• duplicates: MRUNIT-86 Configuration not passed to individual mappers/reducers in mapred MapReducerDriver, PipelineMapReduceDriver
• duplicates: MAPREDUCE-1569 Mock Contexts & Configurations